You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Chris Riccomini <cr...@apache.org> on 2017/09/18 18:19:13 UTC

Airflow 1.9.0 status

Hey all,

An update on the 1.9.0 release. Here are the outstanding PRs that are
slated to be included into 1.9.0:

ISSUE ID     |STATUS    |DESCRIPTION
AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
AIRFLOW-1611 |Open      |Customize logging in Airflow
AIRFLOW-1605 |Reopened  |Fix log source of local loggers
AIRFLOW-1604 |Open      |Rename the logger to log
AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception for @on
AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
AIRFLOW-1015 |Open      |TreeView displayed over task instances
AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for @once
AIRFLOW-976  |Open      |Mark success running task causes it to fail
AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_examples to
AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job to real
AIRFLOW-912  |Open      |Refactor tests and build matrix
AIRFLOW-888  |Open      |Operators should not push XComs by default
AIRFLOW-828  |Open      |Add maximum size for XComs
AIRFLOW-825  |Open      |Add Dataflow semantics
AIRFLOW-788  |Open      |Context unexpectedly added to hive conf

I will be locking down what can get cherry-picked into the 1.9.0 branch
shortly, so if you have something you want in, please set the fix version
to 1.9.0.

We (at WePay) have deployed 1.9.0 into our dev cluster, and it has been
running smoothly for several days.

** I could really use help verifying stability. If you run Airflow, it's in
your best interest to deploy the 1.9.0 test branch somewhere, and verify
it's working for your workload. **

Cheers,
Chris

Re: Airflow 1.9.0 status

Posted by Feng Lu <fe...@google.com.INVALID>.

Thank you so much Chris!

On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <cr...@apache.org>
wrote:

> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
> will be included in alpha1.
>
> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid> wrote:
>
> > Hi Chris,
> >
> > I know it's annoying to have last minute commit com in, but this is a
> > highly desirable feature for folks using GCP operators, is it possible to
> > include AIRFLOW-1635
> > <https://github.com/apache/incubator-airflow/commit/
> > b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4>
> > in?
> > More than happy to offer help if there's something I can do.
> > Thanks a lot.
> >
> > Feng
> >
> > On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> > > Hey all,
> > >
> > > I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
> > >
> > >   https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0alpha0/
> > >
> > > The bin tarball can be installed with pip:
> > >
> > >   pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> > >
> > > The goal is to have the community install and run this to expose any
> bugs
> > > before we move on to official release candidates.
> > >
> > > Here are the outstanding blocker bugs for 1.9.0:
> > >
> > >   AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >   AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > marked
> > > as
> > >   AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> exception
> > > for
> > > @on
> > >   AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> > stdout
> > >   AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> for
> > > @once
> > >   AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> criccomini@apache.org>
> > > wrote:
> > >
> > > > Welp. Work got in the way, so I'll cut the beta on Monday. :)
> > > >
> > > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> > criccomini@apache.org>
> > > > wrote:
> > > >
> > > >> Works for me. Will try and cut a beta before end of week.
> > > >>
> > > >> Blockers for 1.9.0 are:
> > > >>
> > > >> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > > >> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > > marked
> > > >> as
> > > >> AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> > > >>
> > > >>
> > > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi Chris
> > > >>>
> > > >>> Can I suggest releasing a beta? The stable branch is only cut at RC
> > > >>> time. Betas allow us a broader exposure. It also gives us a point
> of
> > > >>> reference.
> > > >>>
> > > >>> In addition the list below are mostly longer standing issues that
> are
> > > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
> 1258,
> > > and
> > > >>> 976 as blocker?
> > > >>>
> > > >>> Cheers
> > > >>> Bolke
> > > >>>
> > > >>> Verstuurd vanaf mijn iPad
> > > >>>
> > > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> > > criccomini@apache.org>
> > > >>> het volgende geschreven:
> > > >>> >
> > > >>> > Hey all,
> > > >>> >
> > > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> release,
> > > but
> > > >>> > seeing as there are several outstanding bugs, I'm going to delay.
> > > Here
> > > >>> are
> > > >>> > the bugs that I'm tracking:
> > > >>> >
> > > >>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > > >>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> are
> > > >>> marked as
> > > >>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> > > exception
> > > >>> for
> > > >>> > @on
> > > >>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> > > >>> stdout
> > > >>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> exception
> > > >>> for @once
> > > >>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
> Email
> > > is
> > > >>> Not
> > > >>> > be
> > > >>> > AIRFLOW-976  |Bug         |Mark success running task causes it to
> > > fail
> > > >>> >
> > > >>> > These are the priority issues. Once they're merged, I'll cut the
> > > >>> > v1-9-stable and beta release.
> > > >>> >
> > > >>> > If you can help clean this up, that would be really appreciated.
> > > >>> >
> > > >>> > Cheers,
> > > >>> > Chris
> > > >>> >
> > > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> > > >>> criccomini@apache.org>
> > > >>> > wrote:
> > > >>> >
> > > >>> >> Marked it for 1.9.0.
> > > >>> >>
> > > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> cjones@simpli.fi
> > >
> > > >>> wrote:
> > > >>> >>>
> > > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> > > >>> callbacks
> > > >>> >>> are not working correctly without emails... Its not a major
> bug,
> > > but
> > > >>> it
> > > >>> >>> does cause us some annoyance in our current deployment.
> > > >>> >>>
> > > >>> >>> Link to Jira:
> > > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> > > >>> >>>
> > > >>> >>> Link to PR:
> > > >>> >>> https://github.com/apache/incubator-airflow/pull/2415
> > > >>> >>>
> > > >>> >>> Thanks!
> > > >>> >>> Charlie Jones
> > > >>> >>>
> > > >>> >>> CHARLIE JONES
> > > >>> >>> Data Engineer
> > > >>> >>> cjones@simpli.fi  |  M: 972.821.7631
> > > >>> >>> __________________________________________________
> > > >>> >>>
> > > >>> >>>
> > > >>> >>> Programmatic Performance.* Localized.*
> > > >>> >>> __________________________________________________
> > > >>> >>>
> > > >>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> > > >>> >>> 800.840.0768  |  www.simpli.fi
> > > >>> >>>
> > > >>> >>>
> > > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> > > >>> criccomini@apache.org>
> > > >>> >>> wrote:
> > > >>> >>>
> > > >>> >>>> Merged.
> > > >>> >>>>
> > > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> > > >>> >>> ryan.buckley@bluecore.com>
> > > >>> >>>> wrote:
> > > >>> >>>>
> > > >>> >>>>> Would it be possible to include AIRFLOW-1587?
> > > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0
> > branch
> > > >>> due
> > > >>> >>> to
> > > >>> >>>>> this issue.
> > > >>> >>>>>
> > > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> > > >>> >>>>>
> > > >>> >>>>> Thanks,
> > > >>> >>>>> Ryan
> > > >>> >>>>>
> > > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> > > >>> >>> <fokko@driesprong.frl
> > > >>> >>>>>
> > > >>> >>>>> wrote:
> > > >>> >>>>>
> > > >>> >>>>>> Hi All,
> > > >>> >>>>>>
> > > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> > > >>> >>>>>>
> > > >>> >>>>>> Currently importing a custom logging configuration is not
> work
> > > (as
> > > >>> >>> far
> > > >>> >>>>> as I
> > > >>> >>>>>> know). Any feedback on the PR would also be appreciated.
> > > >>> >>>>>>
> > > >>> >>>>>> Cheers, Fokko
> > > >>> >>>>>>
> > > >>> >>>>>>
> > > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> > > criccomini@apache.org
> > > >>> >:
> > > >>> >>>>>>
> > > >>> >>>>>>> Done!
> > > >>> >>>>>>>
> > > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > > >>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> > > >>> >>>>>>>
> > > >>> >>>>>>>> Can you slide the aws and emr connection type fix in?
> > > >>> >>>>>>>>
> > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> > > >>> >>>>>>>>
> > > >>> >>>>>>>> It keeps the connection type from getting blanked out on
> > edit
> > > >>> >>> for
> > > >>> >>>>> these
> > > >>> >>>>>>>> types.
> > > >>> >>>>>>>>
> > > >>> >>>>>>>> Thanks,
> > > >>> >>>>>>>> Mike
> > > >>> >>>>>>>>
> > > >>> >>>>>>>>
> > > >>> >>>>>>>>
> > > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > > >>> >>>>> criccomini@apache.org>
> > > >>> >>>>>>>> wrote:
> > > >>> >>>>>>>>>
> > > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
> > to
> > > >>> >>>> these
> > > >>> >>>>>>> fixes!
> > > >>> >>>>>>>>>
> > > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > > >>> >>>>> alex.guziel@airbnb.com
> > > >>> >>>>>> .
> > > >>> >>>>>>>> invalid
> > > >>> >>>>>>>>>> wrote:
> > > >>> >>>>>>>>>
> > > >>> >>>>>>>>>> Can we get this in?
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> > > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> > > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing
> to
> > > >>> >>> have
> > > >>> >>>> in
> > > >>> >>>>>> 1.9.
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > > >>> >>>>>>>> criccomini@apache.org>
> > > >>> >>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> > > >>> >>> Initial
> > > >>> >>>>>>> warning
> > > >>> >>>>>>>>>>> stands, that I will start locking down what can get
> into
> > > >>> >>> 1.9.0
> > > >>> >>>> at
> > > >>> >>>>>>> that
> > > >>> >>>>>>>>>>> point.
> > > >>> >>>>>>>>>>>
> > > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > > >>> >>>>>> bdbruin@gmail.com>
> > > >>> >>>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>>
> > > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a
> particular
> > > >>> >>> fixed
> > > >>> >>>>>> point
> > > >>> >>>>>>> in
> > > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
> > to
> > > a
> > > >>> >>>> git
> > > >>> >>>>>>> pull.
> > > >>> >>>>>>>>>>>>
> > > >>> >>>>>>>>>>>> Bolke
> > > >>> >>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > > >>> >>>>> criccomini@apache.org
> > > >>> >>>>>>>
> > > >>> >>>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly
> > different?
> > > >>> >>>> IIRC,
> > > >>> >>>>>> it's
> > > >>> >>>>>>>>>>>>> basically the same, just no vote, right?
> > > >>> >>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > > >>> >>>>>>> bdbruin@gmail.com>
> > > >>> >>>>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> > away?
> > > >>> >>>>> Isn’t a
> > > >>> >>>>>>>>>> beta
> > > >>> >>>>>>>>>>> a
> > > >>> >>>>>>>>>>>>>> bit smarter?
> > > >>> >>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>> - Bolke
> > > >>> >>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > > >>> >>>>>> criccomini@apache.org
> > > >>> >>>>>>>>
> > > >>> >>>>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> Hey all,
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
> cut
> > > >>> >>> the
> > > >>> >>>>>> stable
> > > >>> >>>>>>>>>>> branch
> > > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> > > >>> >>> stable
> > > >>> >>>>>> branch
> > > >>> >>>>>>>>>> is
> > > >>> >>>>>>>>>>>>>> cut, I
> > > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked
> > > into
> > > >>> >>>> the
> > > >>> >>>>>>>>>> branch,
> > > >>> >>>>>>>>>>>> and
> > > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> > > >>> >>> release
> > > >>> >>>>>> out.
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> Cheers,
> > > >>> >>>>>>>>>>>>>>> Chris
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > >>> >>>>>>>>>>>> criccomini@apache.org
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>> wrote:
> > > >>> >>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> Hey all,
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> > > >>> >>> outstanding
> > > >>> >>>> PRs
> > > >>> >>>>>>> that
> > > >>> >>>>>>>>>>> are
> > > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> > Variable
> > > >>> >>>>>> endpoint
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> > Airflow
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> > > >>> >>> loggers
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
> NOTICE
> > > >>> >>> issue
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> > > >>> >>> unneeded
> > > >>> >>>>> code
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> > HDFS
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> > |airflow/jobs.py:create_dag_ru
> > > >>> >>> n()
> > > >>> >>>>>>>>>> exception
> > > >>> >>>>>>>>>>>> for
> > > >>> >>>>>>>>>>>>>>>> @on
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> > > >>> >>> include
> > > >>> >>>>>> paused
> > > >>> >>>>>>>>>>> DAGs
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
> can
> > > >>> >>> not
> > > >>> >>>> log
> > > >>> >>>>>> to
> > > >>> >>>>>>>>>>> stdout
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> > task
> > > >>> >>>>>> instances
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> > |airflow/jobs.py:manage_slas()
> > > >>> >>>>>> exception
> > > >>> >>>>>>>>>> for
> > > >>> >>>>>>>>>>>>>>>> @once
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> > > >>> >>> causes
> > > >>> >>>> it
> > > >>> >>>>>> to
> > > >>> >>>>>>>>>> fail
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> > > >>> >>>>>> BackfillJobTest.test_backfill_
> > > >>> >>>>>>>>>>>>>> examples
> > > >>> >>>>>>>>>>>>>>>> to
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> > > >>> >>>>>> tests.CoreTest.test_scheduler_
> > > >>> >>>>>>>>>> job
> > > >>> >>>>>>>>>>>> to
> > > >>> >>>>>>>>>>>>>>>> real
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> > > >>> >>> matrix
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> > > >>> >>> XComs
> > > >>> >>>> by
> > > >>> >>>>>>>>>> default
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
> XComs
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > >>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> added
> > to
> > > >>> >>>> hive
> > > >>> >>>>>> conf
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
> > into
> > > >>> >>> the
> > > >>> >>>>>> 1.9.0
> > > >>> >>>>>>>>>>>> branch
> > > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> > please
> > > >>> >>> set
> > > >>> >>>>> the
> > > >>> >>>>>>> fix
> > > >>> >>>>>>>>>>>>>> version
> > > >>> >>>>>>>>>>>>>>>> to 1.9.0.
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> > > cluster,
> > > >>> >>>> and
> > > >>> >>>>> it
> > > >>> >>>>>>> has
> > > >>> >>>>>>>>>>>> been
> > > >>> >>>>>>>>>>>>>>>> running smoothly for several days.
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
> > you
> > > >>> >>> run
> > > >>> >>>>>>>>>> Airflow,
> > > >>> >>>>>>>>>>>> it's
> > > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> > branch
> > > >>> >>>>>> somewhere,
> > > >>> >>>>>>>>>> and
> > > >>> >>>>>>>>>>>>>> verify
> > > >>> >>>>>>>>>>>>>>>> it's working for your workload. **
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>>> Cheers,
> > > >>> >>>>>>>>>>>>>>>> Chris
> > > >>> >>>>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>
> > > >>> >>>>>>>>>>>>
> > > >>> >>>>>>>>>>>
> > > >>> >>>>>>>>>>
> > > >>> >>>>>>>>
> > > >>> >>>>>>>>
> > > >>> >>>>>>>
> > > >>> >>>>>>
> > > >>> >>>>>
> > > >>> >>>>
> > > >>> >>>
> > > >>> >>
> > > >>> >>
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

If anyone wants review those, they can be merged today. 

Bolke

Sent from my iPhone

> On 30 Oct 2017, at 18:58, Chris Riccomini <cr...@apache.org> wrote:
> 
> Hey all,
> 
> Current blockers for 1.9.0:
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1711
> https://issues.apache.org/jira/browse/AIRFLOW-1018
> 
> Cheers,
> Chris
> 
>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <cr...@apache.org> wrote:
>> Hey all,
>> 
>> Talked with @Bolke on Gitter. Here are the currently blocking issues for 1.9.0:
>> 
>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can be False
>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom config on PYTHONPATH
>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck in queued state
>> 
>> PRs are out for them. After these three are merged, I'm planning to cut an RC.
>> 
>> Cheers,
>> Chris
>> 
>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>>> I upgraded our production environment today. Some observations
>>> 
>>> 1. Database migration (add max tries) didn’t workout of the box correctly for Postgres. “max_tries = False” didn’t work and needed to be casted to an int. Not sure how this gets to be false. This is a block imho
>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed to rework quite a lot of dags
>>> 3. We use LdAP for logins. We havent configured the ‘superuser’ ‘data_profiler’ groups and thus should have automatic superuser privileges. We don’t at the moment (eg. we cannot manage connections or see the admin screens).
>>> 
>>> After fixing #1, the rest seems quite normal. Happy with the new logging.
>>> 
>>> Bolke
>>> 
>>> 
>>> > On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>> >
>>> > Hi Bolke,
>>> >
>>> > This one is a blocker: https://issues.apache.org/jira/browse/AIRFLOW-1731.
>>> >
>>> > This requirers that we are sure that the /plugin/ folder is on the path:
>>> > https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>> >
>>> > And maybe add an additional test and updating the docs a bit. Hopefully
>>> > somewhere in the next few days, however I'm a bit busy with the Spark
>>> > Summit.
>>> >
>>> > Cheers, Fokko
>>> >
>>> > 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>> >
>>> >> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>> >>
>>> >> What are the issues that are open?
>>> >>
>>> >> Cheers
>>> >> Bolke
>>> >>
>>> >>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org> wrote:
>>> >>>
>>> >>> K, I will plan on cutting an alpha1 later this week. Something for you
>>> >> guys to play with.
>>> >>>
>>> >>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>>> >> <ma...@gmail.com>> wrote:
>>> >>> hi Chris,
>>> >>>
>>> >>> We are still running pre alpha. I am a bit preoccupied with preparing
>>> >> for a conference and the team is readying a release of one of our core
>>> >> products. So it will probably will be after this week when I get my hands
>>> >> dirty again.
>>> >>>
>>> >>> Cheers
>>> >>> Bolke
>>> >>>
>>> >>> Verstuurd vanaf mijn iPad
>>> >>>
>>> >>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>>> >> <ma...@apache.org>> het volgende geschreven:
>>> >>>>
>>> >>>> Hey all,
>>> >>>>
>>> >>>> Checking in. Has anyone tested this in their environments? Stable
>>> >> releases
>>> >>>> depend on community involvement.
>>> >>>>
>>> >>>> Cheers,
>>> >>>> Chris
>>> >>>>
>>> >>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>> >> criccomini@apache.org <ma...@apache.org>>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>>> >> but
>>> >>>>> will be included in alpha1.
>>> >>>>>
>>> >>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid>
>>> >> wrote:
>>> >>>>>>
>>> >>>>>> Hi Chris,
>>> >>>>>>
>>> >>>>>> I know it's annoying to have last minute commit com in, but this is a
>>> >>>>>> highly desirable feature for folks using GCP operators, is it
>>> >> possible to
>>> >>>>>> include AIRFLOW-1635
>>> >>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>>> >> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>> >>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>> >>>>>> in?
>>> >>>>>> More than happy to offer help if there's something I can do.
>>> >>>>>> Thanks a lot.
>>> >>>>>>
>>> >>>>>> Feng
>>> >>>>>>
>>> >>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>> >> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hey all,
>>> >>>>>>>
>>> >>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>>> >> here:
>>> >>>>>>>
>>> >>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> >> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>> >> airflow/1.9.0alpha0/>
>>> >>>>>>>
>>> >>>>>>> The bin tarball can be installed with pip:
>>> >>>>>>>
>>> >>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>> >>>>>>>
>>> >>>>>>> The goal is to have the community install and run this to expose any
>>> >>>>>> bugs
>>> >>>>>>> before we move on to official release candidates.
>>> >>>>>>>
>>> >>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>> >>>>>>>
>>> >>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> >>>>>> marked
>>> >>>>>>> as
>>> >>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>> >> exception
>>> >>>>>>> for
>>> >>>>>>> @on
>>> >>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>> >>>>>> stdout
>>> >>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>> >>>>>> for
>>> >>>>>>> @once
>>> >>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>> >> fail
>>> >>>>>>>
>>> >>>>>>> Cheers,
>>> >>>>>>> Chris
>>> >>>>>>>
>>> >>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>> >> criccomini@apache.org <ma...@apache.org>
>>> >>>>>>>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>> >>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>> >>>>>>>>>
>>> >>>>>>>>> Blockers for 1.9.0 are:
>>> >>>>>>>>>
>>> >>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>> >>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> >>>>>>> marked
>>> >>>>>>>>> as
>>> >>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>> >>>>>> fail
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>> >> bdbruin@gmail.com <ma...@gmail.com>>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Hi Chris
>>> >>>>>>>>>>
>>> >>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut at
>>> >> RC
>>> >>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>>> >> point of
>>> >>>>>>>>>> reference.
>>> >>>>>>>>>>
>>> >>>>>>>>>> In addition the list below are mostly longer standing issues that
>>> >>>>>> are
>>> >>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>>> >> 1258,
>>> >>>>>>> and
>>> >>>>>>>>>> 976 as blocker?
>>> >>>>>>>>>>
>>> >>>>>>>>>> Cheers
>>> >>>>>>>>>> Bolke
>>> >>>>>>>>>>
>>> >>>>>>>>>> Verstuurd vanaf mijn iPad
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>> >>>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>>>> het volgende geschreven:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>> >>>>>> release,
>>> >>>>>>> but
>>> >>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>>> >> delay.
>>> >>>>>>> Here
>>> >>>>>>>>>> are
>>> >>>>>>>>>>> the bugs that I'm tracking:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>> >>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>> >> are
>>> >>>>>>>>>> marked as
>>> >>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>> >>>>>>> exception
>>> >>>>>>>>>> for
>>> >>>>>>>>>>> @on
>>> >>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>>> >> to
>>> >>>>>>>>>> stdout
>>> >>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>> >>>>>> exception
>>> >>>>>>>>>> for @once
>>> >>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>> >>>>>> Email
>>> >>>>>>> is
>>> >>>>>>>>>> Not
>>> >>>>>>>>>>> be
>>> >>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>> >> to
>>> >>>>>>> fail
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut the
>>> >>>>>>>>>>> v1-9-stable and beta release.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> If you can help clean this up, that would be really appreciated.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Cheers,
>>> >>>>>>>>>>> Chris
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Marked it for 1.9.0.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>> >>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>>> >>>>>>>>>> callbacks
>>> >>>>>>>>>>>>> are not working correctly without emails... Its not a major
>>> >> bug,
>>> >>>>>>> but
>>> >>>>>>>>>> it
>>> >>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Link to Jira:
>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Link to PR:
>>> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>>> >> https://github.com/apache/incubator-airflow/pull/2415>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Thanks!
>>> >>>>>>>>>>>>> Charlie Jones
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> CHARLIE JONES
>>> >>>>>>>>>>>>> Data Engineer
>>> >>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>> >> 972.821.7631 <tel:972.821.7631>
>>> >>>>>>>>>>>>> __________________________________________________
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>> >>>>>>>>>>>>> __________________________________________________
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>> >>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>> >> http://www.simpli.fi/>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Merged.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>> >>>>>>>>>>>>> ryan.buckley@bluecore.com <ma...@bluecore.com>>
>>> >>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>> >>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>> >>>>>> branch
>>> >>>>>>>>>> due
>>> >>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>> this issue.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>>> >> https://github.com/apache/incubator-airflow/pull/2590>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>> Ryan
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>> >>>>>>>>>>>>> <fokko@driesprong.frl
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Hi All,
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>> >>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>>> >> https://github.com/apache/incubator-airflow/pull/2631>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>> >>>>>> work
>>> >>>>>>> (as
>>> >>>>>>>>>>>>> far
>>> >>>>>>>>>>>>>>> as I
>>> >>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Cheers, Fokko
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>> >>>>>>> criccomini@apache.org <ma...@apache.org>
>>> >>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Done!
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>> >>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>> >> michael.crawford@modernizingmedicine.com>> wrote:
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>> >> https://github.com/apache/incubator-airflow/pull/2626> <
>>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>> >> https://github.com/apache/incubator-airflow/pull/2626>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
>>> >>>>>> edit
>>> >>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>>> types.
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>>>>> Mike
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>>> >> forward
>>> >>>>>> to
>>> >>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>> fixes!
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>> >>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>>> >>>>>>>>>>>>>>>> .
>>> >>>>>>>>>>>>>>>>>> invalid
>>> >>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> Can we get this in?
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>>> >> https://github.com/apache/incubator-airflow/commit/>
>>> >>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>>> >> https://github.com/apache/incubator-airflow/commit/>
>>> >>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing
>>> >> to
>>> >>>>>>>>>>>>> have
>>> >>>>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>> 1.9.
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>> >>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>> >>>>>>>>>>>>> Initial
>>> >>>>>>>>>>>>>>>>> warning
>>> >>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>>> >> into
>>> >>>>>>>>>>>>> 1.9.0
>>> >>>>>>>>>>>>>> at
>>> >>>>>>>>>>>>>>>>> that
>>> >>>>>>>>>>>>>>>>>>>>> point.
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>> >>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>> >>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>> >> particular
>>> >>>>>>>>>>>>> fixed
>>> >>>>>>>>>>>>>>>> point
>>> >>>>>>>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>>> >> than
>>> >>>>>> to
>>> >>>>>>> a
>>> >>>>>>>>>>>>>> git
>>> >>>>>>>>>>>>>>>>> pull.
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> Bolke
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>> >>>>>> different?
>>> >>>>>>>>>>>>>> IIRC,
>>> >>>>>>>>>>>>>>>> it's
>>> >>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>> >>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
>>> >>>>>> away?
>>> >>>>>>>>>>>>>>> Isn’t a
>>> >>>>>>>>>>>>>>>>>>>> beta
>>> >>>>>>>>>>>>>>>>>>>>> a
>>> >>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>> >>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
>>> >>>>>> cut
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>> stable
>>> >>>>>>>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>>> >> the
>>> >>>>>>>>>>>>> stable
>>> >>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>> is
>>> >>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>> >>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>>> >> picked
>>> >>>>>>> into
>>> >>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>>>> branch,
>>> >>>>>>>>>>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>>> >> the
>>> >>>>>>>>>>>>> release
>>> >>>>>>>>>>>>>>>> out.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
>>> >> <
>>> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>> >>>>>>>>>>>>> outstanding
>>> >>>>>>>>>>>>>> PRs
>>> >>>>>>>>>>>>>>>>> that
>>> >>>>>>>>>>>>>>>>>>>>> are
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>> >>>>>> Variable
>>> >>>>>>>>>>>>>>>> endpoint
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>> >>>>>> Airflow
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>> >>>>>>>>>>>>> loggers
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>>> >> NOTICE
>>> >>>>>>>>>>>>> issue
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>> >>>>>>>>>>>>> unneeded
>>> >>>>>>>>>>>>>>> code
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
>>> >>>>>> HDFS
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>> >>>>>> |airflow/jobs.py:create_dag_ru
>>> >>>>>>>>>>>>> n()
>>> >>>>>>>>>>>>>>>>>>>> exception
>>> >>>>>>>>>>>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>> >>>>>>>>>>>>> include
>>> >>>>>>>>>>>>>>>> paused
>>> >>>>>>>>>>>>>>>>>>>>> DAGs
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
>>> >>>>>> can
>>> >>>>>>>>>>>>> not
>>> >>>>>>>>>>>>>> log
>>> >>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>> stdout
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
>>> >>>>>> task
>>> >>>>>>>>>>>>>>>> instances
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>> >>>>>> |airflow/jobs.py:manage_slas()
>>> >>>>>>>>>>>>>>>> exception
>>> >>>>>>>>>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>>> >> task
>>> >>>>>>>>>>>>> causes
>>> >>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>> fail
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>> >>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>> >>>>>>>>>>>>>>>>>>>>>>>> examples
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>> >>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>> >>>>>>>>>>>>>>>>>>>> job
>>> >>>>>>>>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> real
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>> >>>>>>>>>>>>> matrix
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>>> >> push
>>> >>>>>>>>>>>>> XComs
>>> >>>>>>>>>>>>>> by
>>> >>>>>>>>>>>>>>>>>>>> default
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>>> >> XComs
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>> >>>>>> added to
>>> >>>>>>>>>>>>>> hive
>>> >>>>>>>>>>>>>>>> conf
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
>>> >>>>>> into
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>> 1.9.0
>>> >>>>>>>>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>> >>>>>> please
>>> >>>>>>>>>>>>> set
>>> >>>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>> fix
>>> >>>>>>>>>>>>>>>>>>>>>>>> version
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>> >>>>>>> cluster,
>>> >>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>>> has
>>> >>>>>>>>>>>>>>>>>>>>>> been
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>>> >> If
>>> >>>>>> you
>>> >>>>>>>>>>>>> run
>>> >>>>>>>>>>>>>>>>>>>> Airflow,
>>> >>>>>>>>>>>>>>>>>>>>>> it's
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>> >>>>>> branch
>>> >>>>>>>>>>>>>>>> somewhere,
>>> >>>>>>>>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>>>>>>>> verify
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>
>>> >>
>>> >>
>>> 
>> 
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

If you make that v1-9-test into a tar we will test with you :-).

> On 3 Nov 2017, at 22:59, Chris Riccomini <cr...@apache.org> wrote:
> 
> Lol, so sorry. It's going to slip to Monday. Today got busy, and we just
> deployed latest v1-9-test to prd. Wanted to let it run over the weekend.
> 
> Sorry for the delay. Monday (or weekend, if I'm motivated).
> 
> On Fri, Nov 3, 2017 at 2:49 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> 
>> Will we get it before the weekend? Oh the suspense ;-)
>> 
>> Sent from my iPhone
>> 
>>> On 2 Nov 2017, at 17:17, Chris Riccomini <cr...@apache.org> wrote:
>>> 
>>> @mC, yes it will.
>>> 
>>> @Bolke, awesome! Will cut an RC today/tomorrow. :) Very exciting.
>>> 
>>> On Thu, Nov 2, 2017 at 8:53 AM, Cieplucha, Michal <
>>> michal.cieplucha@intel.com> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> Will this:
>>>> [AIRFLOW-1641] Handle executor events in the scheduler
>>>> https://github.com/apache/incubator-airflow/commit/
>>>> 2abead7049806482047e29d123a109b444c00355
>>>> 
>>>> be included in 1.9?
>>>> 
>>>> mC
>>>> 
>>>> -----Original Message-----
>>>> From: Bolke de Bruin [mailto:bdbruin@gmail.com]
>>>> Sent: Wednesday, November 1, 2017 11:42 PM
>>>> To: dev@airflow.incubator.apache.org
>>>> Subject: Re: Airflow 1.9.0 status
>>>> 
>>>> I think we are ready for RC.
>>>> 
>>>> Bolke
>>>> 
>>>> 
>>>>> On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org>
>> wrote:
>>>>> 
>>>>> Great, sounds like this is the last remaining JIRA, then?
>>>>> 
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>> 
>>>>> On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> Already fixed.
>>>>>> 
>>>>>>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Is this bug issue on DAG scheduling
>>>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
>>>>>> needs
>>>>>>> resolution before the RC?
>>>>>>> 
>>>>>>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini
>>>>>>> <cr...@apache.org>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Both blockers have been merged. Are we ready for an RC?
>>>>>>>> 
>>>>>>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin
>>>>>>>> <bd...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE).
>>>>>>>>> But
>>>>>> 1765
>>>>>>>>> is not a vulnerability, although *maybe* unexpected behaviour.
>>>>>>>>> 
>>>>>>>>> Bolke.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I marked them as blocker. Still would like to understand how
>>>>>>>>>> important
>>>>>>>>> this
>>>>>>>>>> is, though. The API is experimental, so I could see an argument
>>>>>>>>>> for
>>>>>>>> them
>>>>>>>>>> not being blocker provided the vulnerability is limited to
>>>>>> experimental
>>>>>>>>>> areas.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
>>>>>>>> ash@firemirror.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> I think we need to add
>>>>>>>>>>> 
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>>>>>>>>>>> 
>>>>>>>>>>> to that list as critical security fixes. I'll tackled 1765
>>>>>>>>>>> tomorrow
>>>>>>>> (UK
>>>>>>>>>>> time) if no one gets round to it before then.
>>>>>>>>>>> 
>>>>>>>>>>> -ash
>>>>>>>>>>> 
>>>>>>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini
>>>>>>>>>>>> <cr...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Current blockers for 1.9.0:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
>>>>>> issues
>>>>>>>>> for
>>>>>>>>>>>>> 1.9.0:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
>>>>>> |task.retries
>>>>>>>>> can
>>>>>>>>>>>>> be False
>>>>>>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
>>>>>>>> custom
>>>>>>>>>>>>> config on PYTHONPATH
>>>>>>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task
>>>> gets
>>>>>>>>> stuck
>>>>>>>>>>>>> in queued state
>>>>>>>>>>>>> 
>>>>>>>>>>>>> PRs are out for them. After these three are merged, I'm
>>>>>>>>>>>>> planning to
>>>>>>>>> cut
>>>>>>>>>>> an
>>>>>>>>>>>>> RC.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin
>>>>>>>>>>>>> <bdbruin@gmail.com
>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I upgraded our production environment today. Some
>>>>>>>>>>>>>> observations
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the
>>>>>>>>>>>>>> box
>>>>>>>>>>> correctly
>>>>>>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to
>>>>>>>>>>>>>> be
>>>>>>>> casted
>>>>>>>>>>> to an
>>>>>>>>>>>>>> int. Not sure how this gets to be false. This is a block imho
>>>>>>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility
>>>>>>>>>>>>>> sucks. We
>>>>>>>>>>> needed
>>>>>>>>>>>>>> to rework quite a lot of dags 3. We use LdAP for logins. We
>>>>>>>>>>>>>> havent configured the ‘superuser’
>>>>>>>>>>>>>> ‘data_profiler’ groups and thus should have automatic
>>>>>>>>>>>>>> superuser
>>>>>>>>>>> privileges.
>>>>>>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or
>>>>>>>>>>>>>> see
>>>>>> the
>>>>>>>>>>> admin
>>>>>>>>>>>>>> screens).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the
>>>>>>>>>>>>>> new
>>>>>>>>>>> logging.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko
>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Bolke,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>>>>>>>>>>> /browse/AIRFLOW-1731.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is
>>>>>>>>>>>>>>> on
>>>>>> the
>>>>>>>>>>> path:
>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>>>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with
>>>>>>>>>>>>>>> the
>>>>>>>>> Spark
>>>>>>>>>>>>>>> Summit.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com
>>> :
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX
>> tomorrow.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What are the issues that are open?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
>>>>>> criccomini@apache.org
>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week.
>>>>>>>>>>>>>>>>> Something
>>>>>>>> for
>>>>>>>>>>> you
>>>>>>>>>>>>>>>> guys to play with.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>>>>>> hi Chris,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>> preparing
>>>>>>>>>>>>>>>> for a conference and the team is readying a release of one
>>>>>>>>>>>>>>>> of
>>>>>> our
>>>>>>>>>>> core
>>>>>>>>>>>>>>>> products. So it will probably will be after this week when
>>>>>>>>>>>>>>>> I get
>>>>>>>> my
>>>>>>>>>>>>>> hands
>>>>>>>>>>>>>>>> dirty again.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
>>>>>>>> Stable
>>>>>>>>>>>>>>>> releases
>>>>>>>>>>>>>>>>>> depend on community involvement.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's
>>>>>>>>>>>>>>>>>>> not in
>>>>>>>>>>> alpha0,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>> will be included in alpha1.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>>>>>>>>>>> <fenglu@google.com.invalid
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in,
>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>> this
>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators,
>>>>>>>>>>>>>>>>>>>> is
>>>>>> it
>>>>>>>>>>>>>>>> possible to
>>>>>>>>>>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
>>>>>> b3e985a3
>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>>>>>>>> More than happy to offer help if there's something I can
>>>> do.
>>>>>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
>>>>>>>> download
>>>>>>>>> it
>>>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airfl
>>>>>>>>>>>>>>>>>>>>> ow/1
>>>>>> .
>>>>>>>>>>>>>>>> 9.0alpha0/
>>>>>>>>>>>>>>>> <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>>>>>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
>>>>>>>> incubating-bin.tar.gz
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The goal is to have the community install and run this
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>> expose
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>>>> issue
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>>> SubDagOperator
>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>>>>>> run()
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
>>>> not
>>>>>>>> log
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>> causes
>>>>>>>> it
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on
>>>> Monday.
>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of
>>>> week.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
>> Airflow
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
>>>>>>>>>>>>>>>>>>>>>>> NOTICE
>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>>>> SubDagOperator
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>>>>>> causes
>>>>>>>>> it
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>> only
>>>>>>>>> cut
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> RC
>>>>>>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also
>>>>>>>>>>>>>>>>>>>>>>>> gives
>>>>>>>> us a
>>>>>>>>>>>>>>>> point of
>>>>>>>>>>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer
>>>>>>>>>>>>>>>>>>>>>>>> standing
>>>>>>>>> issues
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider
>>>>>>>>>>>>>>>>>>>>>>>> 1611,
>>>>>>>>>>> 1525,
>>>>>>>>>>>>>>>> 1258,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and
>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm
>>>>>>>>>>>>>>>>>>>>>>>>> going
>>>>>>>> to
>>>>>>>>>>>>>>>> delay.
>>>>>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
>>>>>>>>>>>>>>>>>>>>>>>>> NOTICE
>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>>>>>> SubDagOperator
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug
>>>> |airflow/jobs.py:create_dag_
>>>>>>>>> run()
>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes
>>>> can
>>>>>>>> not
>>>>>>>>>>> log
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
>>>>>>>> Repeated
>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> Email
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running
>> task
>>>>>>>>> causes
>>>>>>>>>>> it
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're
>>>>>>>>>>>>>>>>>>>>>>>>> merged,
>>>>>> I'll
>>>>>>>>> cut
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be
>>>>>>>>>>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini
>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>> 1.9.0?
>>>>>>>>>>>>>> SLA
>>>>>>>>>>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its
>>>>>>>>>>>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>> major
>>>>>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
>>>>>>>> deployment.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-98
>>>>>>>>>>>>>>>>>>>>>>>>>>> 8 <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>> incubator-airflow/pull/2415
>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |
>> M:
>>>>>>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth,
>>>>>>>>>>>>>>>>>>>>>>>>>>> TX
>>>>>>>> 76102
>>>>>>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |
>>>>>>>>>>>>>>>>>>>>>>>>>>> www.simpli.fi <
>>>>>>>>>>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
>>>>>>>> ryan.buckley@bluecore
>>>>>>>>> .
>>>>>>>>>>> com
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the
>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>> incubator-airflow/pull/2590
>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>> incubator-airflow/pull/2631
>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>> is
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Crawford
>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
>>>>>> <mailto:
>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>> fix
>>>>>>>>> in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>> jira/browse/AIRFLOW-1636
>>>>>>>>> <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>> jira/browse/AIRFLOW-1636
>>>>>>>>> <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
>>>>>> blanked
>>>>>>>>> out
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>> criccomini@apache.org
>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
>>>>>>>> looking
>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guziel <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
>>>>>>>> alex.guziel@airbnb.com
>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>> jira/browse/AIRFLOW-1519
>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>> jira/browse/AIRFLOW-1621
>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>>>> incubator-airflow/commit/
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>>>> incubator-airflow/commit/
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>> good
>>>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next
>>>>>>>> week,
>>>>>>>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>> can
>>>>>>>>>>> get
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> de
>>>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a
>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>> tarball
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini
>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>> criccomini@apache.org
>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
>>>>>>>> significantly
>>>>>>>>>>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote,
>> right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke de
>>>>>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>> RCs
>>>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
>>>>>> Riccomini
>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>> planning
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
>>>>>> vote.
>>>>>>>>>>> Once
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>> cherry
>>>>>>>>>>>>>>>> picked
>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required
>>>>>>>> to
>>>>>>>>>>> get
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> Riccomini
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here
>>>>>> are
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS
>> |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
>>>>>>>>> Vulnerability in
>>>>>>>>>>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
>>>>>>>>> logging in
>>>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>> of
>>>>>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
>>>>>>>>> logger to
>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
>>>>>>>>> LICENSE &
>>>>>>>>>>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
>>>>>>>>> duplicate
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open
>> |HDFSOperator
>>>>>> to
>>>>>>>>>>>>>> operate
>>>>>>>>>>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open
>>>> |active_dagruns
>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler
>>>> DAG
>>>>>>>>>>>>>> processes
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
>>>>>>>>> displayed
>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark
>> success
>>>>>>>>> running
>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor
>>>> tests
>>>>>>>>> and
>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
>>>>>>>> should
>>>>>>>>> not
>>>>>>>>>>>>>>>> push
>>>>>>>>>>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add
>> maximum
>>>>>>>> size
>>>>>>>>> for
>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add
>> Dataflow
>>>>>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
>>>>>>>>> unexpectedly
>>>>>>>>>>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>>>>>>>>>>> cherry-picked
>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
>>>>>>>> want
>>>>>>>>>>> in,
>>>>>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>> our
>>>>>>>>>>> dev
>>>>>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verifying
>>>>>>>>>>> stability.
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
>>>>>>>> 1.9.0
>>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> --------------------------------------------------------------------
>>>> 
>>>> Intel Technology Poland sp. z o.o.
>>>> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII
>>>> Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP
>>>> 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>>>> 
>>>> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego
>>>> adresata i moze zawierac informacje poufne. W razie przypadkowego
>>>> otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale
>> jej
>>>> usuniecie; jakiekolwiek
>>>> przegladanie lub rozpowszechnianie jest zabronione.
>>>> This e-mail and any attachments may contain confidential material for
>> the
>>>> sole use of the intended recipient(s). If you are not the intended
>>>> recipient, please contact the sender and delete all copies; any review
>> or
>>>> distribution by
>>>> others is strictly prohibited.
>>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Lol, so sorry. It's going to slip to Monday. Today got busy, and we just
deployed latest v1-9-test to prd. Wanted to let it run over the weekend.

Sorry for the delay. Monday (or weekend, if I'm motivated).

On Fri, Nov 3, 2017 at 2:49 PM, Bolke de Bruin <bd...@gmail.com> wrote:

> Will we get it before the weekend? Oh the suspense ;-)
>
> Sent from my iPhone
>
> > On 2 Nov 2017, at 17:17, Chris Riccomini <cr...@apache.org> wrote:
> >
> > @mC, yes it will.
> >
> > @Bolke, awesome! Will cut an RC today/tomorrow. :) Very exciting.
> >
> > On Thu, Nov 2, 2017 at 8:53 AM, Cieplucha, Michal <
> > michal.cieplucha@intel.com> wrote:
> >
> >> Hello,
> >>
> >> Will this:
> >> [AIRFLOW-1641] Handle executor events in the scheduler
> >> https://github.com/apache/incubator-airflow/commit/
> >> 2abead7049806482047e29d123a109b444c00355
> >>
> >> be included in 1.9?
> >>
> >> mC
> >>
> >> -----Original Message-----
> >> From: Bolke de Bruin [mailto:bdbruin@gmail.com]
> >> Sent: Wednesday, November 1, 2017 11:42 PM
> >> To: dev@airflow.incubator.apache.org
> >> Subject: Re: Airflow 1.9.0 status
> >>
> >> I think we are ready for RC.
> >>
> >> Bolke
> >>
> >>
> >>> On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org>
> wrote:
> >>>
> >>> Great, sounds like this is the last remaining JIRA, then?
> >>>
> >>> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >>>
> >>> On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>
> >>>> Already fixed.
> >>>>
> >>>>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Is this bug issue on DAG scheduling
> >>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
> >>>> needs
> >>>>> resolution before the RC?
> >>>>>
> >>>>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini
> >>>>> <cr...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Both blockers have been merged. Are we ready for an RC?
> >>>>>>
> >>>>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin
> >>>>>> <bd...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE).
> >>>>>>> But
> >>>> 1765
> >>>>>>> is not a vulnerability, although *maybe* unexpected behaviour.
> >>>>>>>
> >>>>>>> Bolke.
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> I marked them as blocker. Still would like to understand how
> >>>>>>>> important
> >>>>>>> this
> >>>>>>>> is, though. The API is experimental, so I could see an argument
> >>>>>>>> for
> >>>>>> them
> >>>>>>>> not being blocker provided the vulnerability is limited to
> >>>> experimental
> >>>>>>>> areas.
> >>>>>>>>
> >>>>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
> >>>>>> ash@firemirror.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I think we need to add
> >>>>>>>>>
> >>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
> >>>>>>>>>
> >>>>>>>>> to that list as critical security fixes. I'll tackled 1765
> >>>>>>>>> tomorrow
> >>>>>> (UK
> >>>>>>>>> time) if no one gets round to it before then.
> >>>>>>>>>
> >>>>>>>>> -ash
> >>>>>>>>>
> >>>>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini
> >>>>>>>>>> <cr...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hey all,
> >>>>>>>>>>
> >>>>>>>>>> Current blockers for 1.9.0:
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Chris
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> >>>>>>> criccomini@apache.org
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey all,
> >>>>>>>>>>>
> >>>>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
> >>>> issues
> >>>>>>> for
> >>>>>>>>>>> 1.9.0:
> >>>>>>>>>>>
> >>>>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
> >>>> |task.retries
> >>>>>>> can
> >>>>>>>>>>> be False
> >>>>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
> >>>>>> custom
> >>>>>>>>>>> config on PYTHONPATH
> >>>>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task
> >> gets
> >>>>>>> stuck
> >>>>>>>>>>> in queued state
> >>>>>>>>>>>
> >>>>>>>>>>> PRs are out for them. After these three are merged, I'm
> >>>>>>>>>>> planning to
> >>>>>>> cut
> >>>>>>>>> an
> >>>>>>>>>>> RC.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Chris
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin
> >>>>>>>>>>> <bdbruin@gmail.com
> >>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I upgraded our production environment today. Some
> >>>>>>>>>>>> observations
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the
> >>>>>>>>>>>> box
> >>>>>>>>> correctly
> >>>>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to
> >>>>>>>>>>>> be
> >>>>>> casted
> >>>>>>>>> to an
> >>>>>>>>>>>> int. Not sure how this gets to be false. This is a block imho
> >>>>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility
> >>>>>>>>>>>> sucks. We
> >>>>>>>>> needed
> >>>>>>>>>>>> to rework quite a lot of dags 3. We use LdAP for logins. We
> >>>>>>>>>>>> havent configured the ‘superuser’
> >>>>>>>>>>>> ‘data_profiler’ groups and thus should have automatic
> >>>>>>>>>>>> superuser
> >>>>>>>>> privileges.
> >>>>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or
> >>>>>>>>>>>> see
> >>>> the
> >>>>>>>>> admin
> >>>>>>>>>>>> screens).
> >>>>>>>>>>>>
> >>>>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the
> >>>>>>>>>>>> new
> >>>>>>>>> logging.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Bolke
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko
> >>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Bolke,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
> >>>>>>>>>>>> /browse/AIRFLOW-1731.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is
> >>>>>>>>>>>>> on
> >>>> the
> >>>>>>>>> path:
> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>>>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
> >>>>>>>>> Hopefully
> >>>>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with
> >>>>>>>>>>>>> the
> >>>>>>> Spark
> >>>>>>>>>>>>> Summit.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com
> >:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX
> tomorrow.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What are the issues that are open?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week.
> >>>>>>>>>>>>>>> Something
> >>>>>> for
> >>>>>>>>> you
> >>>>>>>>>>>>>> guys to play with.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com
> >>>>>>>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>>>>>>> hi Chris,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied
> >>>>>>>>>>>>>>> with
> >>>>>>>>> preparing
> >>>>>>>>>>>>>> for a conference and the team is readying a release of one
> >>>>>>>>>>>>>> of
> >>>> our
> >>>>>>>>> core
> >>>>>>>>>>>>>> products. So it will probably will be after this week when
> >>>>>>>>>>>>>> I get
> >>>>>> my
> >>>>>>>>>>>> hands
> >>>>>>>>>>>>>> dirty again.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> >>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
> >>>>>> Stable
> >>>>>>>>>>>>>> releases
> >>>>>>>>>>>>>>>> depend on community involvement.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's
> >>>>>>>>>>>>>>>>> not in
> >>>>>>>>> alpha0,
> >>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> will be included in alpha1.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> >>>>>>>>> <fenglu@google.com.invalid
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in,
> >>>>>>>>>>>>>>>>>> but
> >>>>>>> this
> >>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators,
> >>>>>>>>>>>>>>>>>> is
> >>>> it
> >>>>>>>>>>>>>> possible to
> >>>>>>>>>>>>>>>>>> include AIRFLOW-1635
> >>>>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
> >>>> b3e985a3
> >>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>>>>>>>>>>> in?
> >>>>>>>>>>>>>>>>>> More than happy to offer help if there's something I can
> >> do.
> >>>>>>>>>>>>>>>>>> Thanks a lot.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
> >>>>>> download
> >>>>>>> it
> >>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airfl
> >>>>>>>>>>>>>>>>>>> ow/1
> >>>> .
> >>>>>>>>>>>>>> 9.0alpha0/
> >>>>>>>>>>>>>> <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>>>>>>>>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
> >>>>>> incubating-bin.tar.gz
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The goal is to have the community install and run this
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>> expose
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>> bugs
> >>>>>>>>>>>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> >> issue
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>>> SubDagOperator
> >>>>>>>>> are
> >>>>>>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> >>>> run()
> >>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
> >> not
> >>>>>> log
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>  |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >> causes
> >>>>>> it
> >>>>>>>>> to
> >>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on
> >> Monday.
> >>>>>> :)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of
> >> week.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
> Airflow
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
> >>>>>>>>>>>>>>>>>>>>> NOTICE
> >>>>>> issue
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>>>> SubDagOperator
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >>>>>> causes
> >>>>>>> it
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>> only
> >>>>>>> cut
> >>>>>>>>>>>> at
> >>>>>>>>>>>>>> RC
> >>>>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also
> >>>>>>>>>>>>>>>>>>>>>> gives
> >>>>>> us a
> >>>>>>>>>>>>>> point of
> >>>>>>>>>>>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer
> >>>>>>>>>>>>>>>>>>>>>> standing
> >>>>>>> issues
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider
> >>>>>>>>>>>>>>>>>>>>>> 1611,
> >>>>>>>>> 1525,
> >>>>>>>>>>>>>> 1258,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and
> >>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>> beta
> >>>>>>>>>>>>>>>>>> release,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm
> >>>>>>>>>>>>>>>>>>>>>>> going
> >>>>>> to
> >>>>>>>>>>>>>> delay.
> >>>>>>>>>>>>>>>>>>> Here
> >>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
> >> Airflow
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
> >>>>>>>>>>>>>>>>>>>>>>> NOTICE
> >>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>>>>>> SubDagOperator
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug
> >> |airflow/jobs.py:create_dag_
> >>>>>>> run()
> >>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes
> >> can
> >>>>>> not
> >>>>>>>>> log
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
> >>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
> >>>>>> Repeated
> >>>>>>>>> if
> >>>>>>>>>>>>>>>>>> Email
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> Not
> >>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running
> task
> >>>>>>> causes
> >>>>>>>>> it
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're
> >>>>>>>>>>>>>>>>>>>>>>> merged,
> >>>> I'll
> >>>>>>> cut
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be
> >>>>>>>>>>>>>>>>>>>>>>> really
> >>>>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini
> >>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988
> >>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>> 1.9.0?
> >>>>>>>>>>>> SLA
> >>>>>>>>>>>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its
> >>>>>>>>>>>>>>>>>>>>>>>>> not a
> >>>>>>>>> major
> >>>>>>>>>>>>>> bug,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
> >>>>>> deployment.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-98
> >>>>>>>>>>>>>>>>>>>>>>>>> 8 <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>> incubator-airflow/pull/2415
> >>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |
> M:
> >>>>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>>> __
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
> >>>>>>>>>>>>>>>>>>>>>>>>> __
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth,
> >>>>>>>>>>>>>>>>>>>>>>>>> TX
> >>>>>> 76102
> >>>>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |
> >>>>>>>>>>>>>>>>>>>>>>>>> www.simpli.fi <
> >>>>>>>>>>>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
> >>>>>> ryan.buckley@bluecore
> >>>>>>> .
> >>>>>>>>> com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken
> >>>>>>>>>>>>>>>>>>>>>>>>>>> on the
> >>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>> incubator-airflow/pull/2590
> >>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
> >>>>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>> release:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>> incubator-airflow/pull/2631
> >>>>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>> is
> >>>>>>>>> not
> >>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>> (as
> >>>>>>>>>>>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> >>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Crawford
> >>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
> >>>> <mailto:
> >>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
> >>>> fix
> >>>>>>> in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>>> jira/browse/AIRFLOW-1636
> >>>>>>> <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>>> jira/browse/AIRFLOW-1636
> >>>>>>> <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>>>> incubator-airflow/pull/2626
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>>>> incubator-airflow/pull/2626
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
> >>>> blanked
> >>>>>>> out
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>> edit
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>> criccomini@apache.org
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
> >>>>>> looking
> >>>>>>>>>>>>>> forward
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guziel <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
> >>>>>> alex.guziel@airbnb.com
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>>>> jira/browse/AIRFLOW-1519
> >>>>>>>>> <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>>>> jira/browse/AIRFLOW-1621
> >>>>>>>>> <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>>>> incubator-airflow/commit/
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>>>> incubator-airflow/commit/
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>> good
> >>>>>>>>>>>> thing
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> >>>>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>>>> criccomini@apache.org>
> >>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next
> >>>>>> week,
> >>>>>>>>>>>> then.
> >>>>>>>>>>>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
> >>>>>> can
> >>>>>>>>> get
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> de
> >>>>>>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a
> >>>>>>>>>>>>>> particular
> >>>>>>>>>>>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>> tarball
> >>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini
> >>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>> criccomini@apache.org
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
> >>>>>> significantly
> >>>>>>>>>>>>>>>>>> different?
> >>>>>>>>>>>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote,
> right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke de
> >>>>>>>>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>>>>> RCs
> >>>>>>>>>>>> right
> >>>>>>>>>>>>>>>>>> away?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
> >>>> Riccomini
> >>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >>>>>>> planning
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
> >>>> vote.
> >>>>>>>>> Once
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
> >>>>>> cherry
> >>>>>>>>>>>>>> picked
> >>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required
> >>>>>> to
> >>>>>>>>> get
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>> Riccomini
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>>>>>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here
> >>>> are
> >>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS
> |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> >>>>>>> Vulnerability in
> >>>>>>>>>>>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> >>>>>>> logging in
> >>>>>>>>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
> >>>>>> of
> >>>>>>>>>>>> local
> >>>>>>>>>>>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> >>>>>>> logger to
> >>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> >>>>>>> LICENSE &
> >>>>>>>>>>>>>> NOTICE
> >>>>>>>>>>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> >>>>>>> duplicate
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open
> |HDFSOperator
> >>>> to
> >>>>>>>>>>>> operate
> >>>>>>>>>>>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open
> >> |active_dagruns
> >>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler
> >> DAG
> >>>>>>>>>>>> processes
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> >>>>>>> displayed
> >>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark
> success
> >>>>>>> running
> >>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor
> >> tests
> >>>>>>> and
> >>>>>>>>>>>> build
> >>>>>>>>>>>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
> >>>>>> should
> >>>>>>> not
> >>>>>>>>>>>>>> push
> >>>>>>>>>>>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add
> maximum
> >>>>>> size
> >>>>>>> for
> >>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add
> Dataflow
> >>>>>>>>> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> >>>>>>> unexpectedly
> >>>>>>>>>>>>>>>>>> added to
> >>>>>>>>>>>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>>>>>>>>>>> cherry-picked
> >>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
> >>>>>> want
> >>>>>>>>> in,
> >>>>>>>>>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
> >>>>>> our
> >>>>>>>>> dev
> >>>>>>>>>>>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verifying
> >>>>>>>>> stability.
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
> >>>>>> 1.9.0
> >>>>>>>>> test
> >>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
> >> --------------------------------------------------------------------
> >>
> >> Intel Technology Poland sp. z o.o.
> >> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII
> >> Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP
> >> 957-07-52-316 | Kapital zakladowy 200.000 PLN.
> >>
> >> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego
> >> adresata i moze zawierac informacje poufne. W razie przypadkowego
> >> otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale
> jej
> >> usuniecie; jakiekolwiek
> >> przegladanie lub rozpowszechnianie jest zabronione.
> >> This e-mail and any attachments may contain confidential material for
> the
> >> sole use of the intended recipient(s). If you are not the intended
> >> recipient, please contact the sender and delete all copies; any review
> or
> >> distribution by
> >> others is strictly prohibited.
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

Will we get it before the weekend? Oh the suspense ;-)

Sent from my iPhone

> On 2 Nov 2017, at 17:17, Chris Riccomini <cr...@apache.org> wrote:
> 
> @mC, yes it will.
> 
> @Bolke, awesome! Will cut an RC today/tomorrow. :) Very exciting.
> 
> On Thu, Nov 2, 2017 at 8:53 AM, Cieplucha, Michal <
> michal.cieplucha@intel.com> wrote:
> 
>> Hello,
>> 
>> Will this:
>> [AIRFLOW-1641] Handle executor events in the scheduler
>> https://github.com/apache/incubator-airflow/commit/
>> 2abead7049806482047e29d123a109b444c00355
>> 
>> be included in 1.9?
>> 
>> mC
>> 
>> -----Original Message-----
>> From: Bolke de Bruin [mailto:bdbruin@gmail.com]
>> Sent: Wednesday, November 1, 2017 11:42 PM
>> To: dev@airflow.incubator.apache.org
>> Subject: Re: Airflow 1.9.0 status
>> 
>> I think we are ready for RC.
>> 
>> Bolke
>> 
>> 
>>> On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org> wrote:
>>> 
>>> Great, sounds like this is the last remaining JIRA, then?
>>> 
>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>> 
>>> On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>> 
>>>> Already fixed.
>>>> 
>>>>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
>>>> wrote:
>>>>> 
>>>>> Is this bug issue on DAG scheduling
>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
>>>> needs
>>>>> resolution before the RC?
>>>>> 
>>>>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini
>>>>> <cr...@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Both blockers have been merged. Are we ready for an RC?
>>>>>> 
>>>>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin
>>>>>> <bd...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE).
>>>>>>> But
>>>> 1765
>>>>>>> is not a vulnerability, although *maybe* unexpected behaviour.
>>>>>>> 
>>>>>>> Bolke.
>>>>>>> 
>>>>>>> 
>>>>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> I marked them as blocker. Still would like to understand how
>>>>>>>> important
>>>>>>> this
>>>>>>>> is, though. The API is experimental, so I could see an argument
>>>>>>>> for
>>>>>> them
>>>>>>>> not being blocker provided the vulnerability is limited to
>>>> experimental
>>>>>>>> areas.
>>>>>>>> 
>>>>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
>>>>>> ash@firemirror.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I think we need to add
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>>>>>>>>> 
>>>>>>>>> to that list as critical security fixes. I'll tackled 1765
>>>>>>>>> tomorrow
>>>>>> (UK
>>>>>>>>> time) if no one gets round to it before then.
>>>>>>>>> 
>>>>>>>>> -ash
>>>>>>>>> 
>>>>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini
>>>>>>>>>> <cr...@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hey all,
>>>>>>>>>> 
>>>>>>>>>> Current blockers for 1.9.0:
>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>>>>> 
>>>>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
>>>>>>> criccomini@apache.org
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey all,
>>>>>>>>>>> 
>>>>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
>>>> issues
>>>>>>> for
>>>>>>>>>>> 1.9.0:
>>>>>>>>>>> 
>>>>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
>>>> |task.retries
>>>>>>> can
>>>>>>>>>>> be False
>>>>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
>>>>>> custom
>>>>>>>>>>> config on PYTHONPATH
>>>>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task
>> gets
>>>>>>> stuck
>>>>>>>>>>> in queued state
>>>>>>>>>>> 
>>>>>>>>>>> PRs are out for them. After these three are merged, I'm
>>>>>>>>>>> planning to
>>>>>>> cut
>>>>>>>>> an
>>>>>>>>>>> RC.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin
>>>>>>>>>>> <bdbruin@gmail.com
>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I upgraded our production environment today. Some
>>>>>>>>>>>> observations
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the
>>>>>>>>>>>> box
>>>>>>>>> correctly
>>>>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to
>>>>>>>>>>>> be
>>>>>> casted
>>>>>>>>> to an
>>>>>>>>>>>> int. Not sure how this gets to be false. This is a block imho
>>>>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility
>>>>>>>>>>>> sucks. We
>>>>>>>>> needed
>>>>>>>>>>>> to rework quite a lot of dags 3. We use LdAP for logins. We
>>>>>>>>>>>> havent configured the ‘superuser’
>>>>>>>>>>>> ‘data_profiler’ groups and thus should have automatic
>>>>>>>>>>>> superuser
>>>>>>>>> privileges.
>>>>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or
>>>>>>>>>>>> see
>>>> the
>>>>>>>>> admin
>>>>>>>>>>>> screens).
>>>>>>>>>>>> 
>>>>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the
>>>>>>>>>>>> new
>>>>>>>>> logging.
>>>>>>>>>>>> 
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko
>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Bolke,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>>>>>>>>> /browse/AIRFLOW-1731.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is
>>>>>>>>>>>>> on
>>>> the
>>>>>>>>> path:
>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>>>>>>>>> 
>>>>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
>>>>>>>>> Hopefully
>>>>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with
>>>>>>>>>>>>> the
>>>>>>> Spark
>>>>>>>>>>>>> Summit.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What are the issues that are open?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
>>>> criccomini@apache.org
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week.
>>>>>>>>>>>>>>> Something
>>>>>> for
>>>>>>>>> you
>>>>>>>>>>>>>> guys to play with.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>>>> hi Chris,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied
>>>>>>>>>>>>>>> with
>>>>>>>>> preparing
>>>>>>>>>>>>>> for a conference and the team is readying a release of one
>>>>>>>>>>>>>> of
>>>> our
>>>>>>>>> core
>>>>>>>>>>>>>> products. So it will probably will be after this week when
>>>>>>>>>>>>>> I get
>>>>>> my
>>>>>>>>>>>> hands
>>>>>>>>>>>>>> dirty again.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
>>>>>> Stable
>>>>>>>>>>>>>> releases
>>>>>>>>>>>>>>>> depend on community involvement.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's
>>>>>>>>>>>>>>>>> not in
>>>>>>>>> alpha0,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> will be included in alpha1.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>>>>>>>>> <fenglu@google.com.invalid
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in,
>>>>>>>>>>>>>>>>>> but
>>>>>>> this
>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators,
>>>>>>>>>>>>>>>>>> is
>>>> it
>>>>>>>>>>>>>> possible to
>>>>>>>>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
>>>> b3e985a3
>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>>>>>> More than happy to offer help if there's something I can
>> do.
>>>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
>>>>>> download
>>>>>>> it
>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airfl
>>>>>>>>>>>>>>>>>>> ow/1
>>>> .
>>>>>>>>>>>>>> 9.0alpha0/
>>>>>>>>>>>>>> <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>>>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
>>>>>> incubating-bin.tar.gz
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The goal is to have the community install and run this
>>>>>>>>>>>>>>>>>>> to
>>>>>>> expose
>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>> issue
>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>> SubDagOperator
>>>>>>>>> are
>>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>>>> run()
>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
>> not
>>>>>> log
>>>>>>>>> to
>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>> causes
>>>>>> it
>>>>>>>>> to
>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on
>> Monday.
>>>>>> :)
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of
>> week.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
>>>>>>>>>>>>>>>>>>>>> NOTICE
>>>>>> issue
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>> SubDagOperator
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>>>> causes
>>>>>>> it
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch
>>>>>>>>>>>>>>>>>>>>>> is
>>>> only
>>>>>>> cut
>>>>>>>>>>>> at
>>>>>>>>>>>>>> RC
>>>>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also
>>>>>>>>>>>>>>>>>>>>>> gives
>>>>>> us a
>>>>>>>>>>>>>> point of
>>>>>>>>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer
>>>>>>>>>>>>>>>>>>>>>> standing
>>>>>>> issues
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider
>>>>>>>>>>>>>>>>>>>>>> 1611,
>>>>>>>>> 1525,
>>>>>>>>>>>>>> 1258,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and
>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>> beta
>>>>>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm
>>>>>>>>>>>>>>>>>>>>>>> going
>>>>>> to
>>>>>>>>>>>>>> delay.
>>>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
>> Airflow
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
>>>>>>>>>>>>>>>>>>>>>>> NOTICE
>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>>>> SubDagOperator
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug
>> |airflow/jobs.py:create_dag_
>>>>>>> run()
>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes
>> can
>>>>>> not
>>>>>>>>> log
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
>>>>>> Repeated
>>>>>>>>> if
>>>>>>>>>>>>>>>>>> Email
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>>>>> causes
>>>>>>>>> it
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're
>>>>>>>>>>>>>>>>>>>>>>> merged,
>>>> I'll
>>>>>>> cut
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be
>>>>>>>>>>>>>>>>>>>>>>> really
>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>> 1.9.0?
>>>>>>>>>>>> SLA
>>>>>>>>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its
>>>>>>>>>>>>>>>>>>>>>>>>> not a
>>>>>>>>> major
>>>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
>>>>>> deployment.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-98
>>>>>>>>>>>>>>>>>>>>>>>>> 8 <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>> incubator-airflow/pull/2415
>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth,
>>>>>>>>>>>>>>>>>>>>>>>>> TX
>>>>>> 76102
>>>>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |
>>>>>>>>>>>>>>>>>>>>>>>>> www.simpli.fi <
>>>>>>>>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris
>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
>>>>>> ryan.buckley@bluecore
>>>>>>> .
>>>>>>>>> com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken
>>>>>>>>>>>>>>>>>>>>>>>>>>> on the
>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>> incubator-airflow/pull/2590
>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/pull/2631
>>>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging
>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>> is
>>>>>>>>> not
>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Crawford
>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
>>>> <mailto:
>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>> fix
>>>>>>> in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>> jira/browse/AIRFLOW-1636
>>>>>>> <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>> jira/browse/AIRFLOW-1636
>>>>>>> <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
>>>> blanked
>>>>>>> out
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>> criccomini@apache.org
>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
>>>>>> looking
>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guziel <
>>>>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
>>>>>> alex.guziel@airbnb.com
>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>> jira/browse/AIRFLOW-1519
>>>>>>>>> <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>> jira/browse/AIRFLOW-1621
>>>>>>>>> <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>> incubator-airflow/commit/
>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>>>> incubator-airflow/commit/
>>>>>>>>>>>> <
>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>> good
>>>>>>>>>>>> thing
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>> criccomini@apache.org>
>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next
>>>>>> week,
>>>>>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>> can
>>>>>>>>> get
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> de
>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a
>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>> tarball
>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini
>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>> criccomini@apache.org
>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
>>>>>> significantly
>>>>>>>>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke de
>>>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>> RCs
>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
>>>> Riccomini
>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>> planning
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
>>>> vote.
>>>>>>>>> Once
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>> cherry
>>>>>>>>>>>>>> picked
>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required
>>>>>> to
>>>>>>>>> get
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>> Riccomini
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here
>>>> are
>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
>>>>>>> Vulnerability in
>>>>>>>>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
>>>>>>> logging in
>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>>>> of
>>>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
>>>>>>> logger to
>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
>>>>>>> LICENSE &
>>>>>>>>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
>>>>>>> duplicate
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
>>>> to
>>>>>>>>>>>> operate
>>>>>>>>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open
>> |active_dagruns
>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler
>> DAG
>>>>>>>>>>>> processes
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
>>>>>>> displayed
>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
>>>>>>> running
>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor
>> tests
>>>>>>> and
>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
>>>>>> should
>>>>>>> not
>>>>>>>>>>>>>> push
>>>>>>>>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
>>>>>> size
>>>>>>> for
>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
>>>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
>>>>>>> unexpectedly
>>>>>>>>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>>>>>>>>> cherry-picked
>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
>>>>>> want
>>>>>>>>> in,
>>>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>> our
>>>>>>>>> dev
>>>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verifying
>>>>>>>>> stability.
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
>>>>>> 1.9.0
>>>>>>>>> test
>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
>> --------------------------------------------------------------------
>> 
>> Intel Technology Poland sp. z o.o.
>> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII
>> Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP
>> 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>> 
>> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego
>> adresata i moze zawierac informacje poufne. W razie przypadkowego
>> otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej
>> usuniecie; jakiekolwiek
>> przegladanie lub rozpowszechnianie jest zabronione.
>> This e-mail and any attachments may contain confidential material for the
>> sole use of the intended recipient(s). If you are not the intended
>> recipient, please contact the sender and delete all copies; any review or
>> distribution by
>> others is strictly prohibited.
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

@mC, yes it will.

@Bolke, awesome! Will cut an RC today/tomorrow. :) Very exciting.

On Thu, Nov 2, 2017 at 8:53 AM, Cieplucha, Michal <
michal.cieplucha@intel.com> wrote:

> Hello,
>
> Will this:
> [AIRFLOW-1641] Handle executor events in the scheduler
> https://github.com/apache/incubator-airflow/commit/
> 2abead7049806482047e29d123a109b444c00355
>
> be included in 1.9?
>
> mC
>
> -----Original Message-----
> From: Bolke de Bruin [mailto:bdbruin@gmail.com]
> Sent: Wednesday, November 1, 2017 11:42 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow 1.9.0 status
>
> I think we are ready for RC.
>
> Bolke
>
>
> > On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org> wrote:
> >
> > Great, sounds like this is the last remaining JIRA, then?
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-1764
> >
> > On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> >> Already fixed.
> >>
> >>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
> >> wrote:
> >>>
> >>> Is this bug issue on DAG scheduling
> >>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
> >> needs
> >>> resolution before the RC?
> >>>
> >>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini
> >>> <cr...@apache.org>
> >>> wrote:
> >>>
> >>>> Both blockers have been merged. Are we ready for an RC?
> >>>>
> >>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin
> >>>> <bd...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE).
> >>>>> But
> >> 1765
> >>>>> is not a vulnerability, although *maybe* unexpected behaviour.
> >>>>>
> >>>>> Bolke.
> >>>>>
> >>>>>
> >>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
> >>>> wrote:
> >>>>>>
> >>>>>> I marked them as blocker. Still would like to understand how
> >>>>>> important
> >>>>> this
> >>>>>> is, though. The API is experimental, so I could see an argument
> >>>>>> for
> >>>> them
> >>>>>> not being blocker provided the vulnerability is limited to
> >> experimental
> >>>>>> areas.
> >>>>>>
> >>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
> >>>> ash@firemirror.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I think we need to add
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
> >>>>>>>
> >>>>>>> to that list as critical security fixes. I'll tackled 1765
> >>>>>>> tomorrow
> >>>> (UK
> >>>>>>> time) if no one gets round to it before then.
> >>>>>>>
> >>>>>>> -ash
> >>>>>>>
> >>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini
> >>>>>>>> <cr...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hey all,
> >>>>>>>>
> >>>>>>>> Current blockers for 1.9.0:
> >>>>>>>>
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> >>>>> criccomini@apache.org
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hey all,
> >>>>>>>>>
> >>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
> >> issues
> >>>>> for
> >>>>>>>>> 1.9.0:
> >>>>>>>>>
> >>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
> >> |task.retries
> >>>>> can
> >>>>>>>>> be False
> >>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
> >>>> custom
> >>>>>>>>> config on PYTHONPATH
> >>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task
> gets
> >>>>> stuck
> >>>>>>>>> in queued state
> >>>>>>>>>
> >>>>>>>>> PRs are out for them. After these three are merged, I'm
> >>>>>>>>> planning to
> >>>>> cut
> >>>>>>> an
> >>>>>>>>> RC.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin
> >>>>>>>>> <bdbruin@gmail.com
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I upgraded our production environment today. Some
> >>>>>>>>>> observations
> >>>>>>>>>>
> >>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the
> >>>>>>>>>> box
> >>>>>>> correctly
> >>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to
> >>>>>>>>>> be
> >>>> casted
> >>>>>>> to an
> >>>>>>>>>> int. Not sure how this gets to be false. This is a block imho
> >>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility
> >>>>>>>>>> sucks. We
> >>>>>>> needed
> >>>>>>>>>> to rework quite a lot of dags 3. We use LdAP for logins. We
> >>>>>>>>>> havent configured the ‘superuser’
> >>>>>>>>>> ‘data_profiler’ groups and thus should have automatic
> >>>>>>>>>> superuser
> >>>>>>> privileges.
> >>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or
> >>>>>>>>>> see
> >> the
> >>>>>>> admin
> >>>>>>>>>> screens).
> >>>>>>>>>>
> >>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the
> >>>>>>>>>> new
> >>>>>>> logging.
> >>>>>>>>>>
> >>>>>>>>>> Bolke
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko
> >>>>>>>>>>> <fokko@driesprong.frl
> >>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Bolke,
> >>>>>>>>>>>
> >>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
> >>>>>>>>>> /browse/AIRFLOW-1731.
> >>>>>>>>>>>
> >>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is
> >>>>>>>>>>> on
> >> the
> >>>>>>> path:
> >>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>>>>>>>>
> >>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
> >>>>>>> Hopefully
> >>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with
> >>>>>>>>>>> the
> >>>>> Spark
> >>>>>>>>>>> Summit.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>
> >>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> >>>>>>>>>>>
> >>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What are the issues that are open?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>> Bolke
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
> >> criccomini@apache.org
> >>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week.
> >>>>>>>>>>>>> Something
> >>>> for
> >>>>>>> you
> >>>>>>>>>>>> guys to play with.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> >>>>> bdbruin@gmail.com
> >>>>>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>>>>> hi Chris,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied
> >>>>>>>>>>>>> with
> >>>>>>> preparing
> >>>>>>>>>>>> for a conference and the team is readying a release of one
> >>>>>>>>>>>> of
> >> our
> >>>>>>> core
> >>>>>>>>>>>> products. So it will probably will be after this week when
> >>>>>>>>>>>> I get
> >>>> my
> >>>>>>>>>> hands
> >>>>>>>>>>>> dirty again.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> >>>>>>> criccomini@apache.org
> >>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
> >>>> Stable
> >>>>>>>>>>>> releases
> >>>>>>>>>>>>>> depend on community involvement.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's
> >>>>>>>>>>>>>>> not in
> >>>>>>> alpha0,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>>> will be included in alpha1.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> >>>>>>> <fenglu@google.com.invalid
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in,
> >>>>>>>>>>>>>>>> but
> >>>>> this
> >>>>>>>>>> is a
> >>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators,
> >>>>>>>>>>>>>>>> is
> >> it
> >>>>>>>>>>>> possible to
> >>>>>>>>>>>>>>>> include AIRFLOW-1635
> >>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
> >> b3e985a3
> >>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3
> >>>>>>>>>>>> >
> >>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>>>>>>>>> in?
> >>>>>>>>>>>>>>>> More than happy to offer help if there's something I can
> do.
> >>>>>>>>>>>>>>>> Thanks a lot.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
> >>>> download
> >>>>> it
> >>>>>>>>>>>> here:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airfl
> >>>>>>>>>>>>>>>>> ow/1
> >> .
> >>>>>>>>>>>> 9.0alpha0/
> >>>>>>>>>>>> <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>>>>>>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
> >>>> incubating-bin.tar.gz
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The goal is to have the community install and run this
> >>>>>>>>>>>>>>>>> to
> >>>>> expose
> >>>>>>>>>> any
> >>>>>>>>>>>>>>>> bugs
> >>>>>>>>>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> issue
> >>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>> SubDagOperator
> >>>>>>> are
> >>>>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> >> run()
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
> not
> >>>> log
> >>>>>>> to
> >>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>>>>>> exception
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> causes
> >>>> it
> >>>>>>> to
> >>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on
> Monday.
> >>>> :)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of
> week.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
> >>>>>>>>>>>>>>>>>>> NOTICE
> >>>> issue
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>> SubDagOperator
> >>>>>>>>>> are
> >>>>>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >>>> causes
> >>>>> it
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch
> >>>>>>>>>>>>>>>>>>>> is
> >> only
> >>>>> cut
> >>>>>>>>>> at
> >>>>>>>>>>>> RC
> >>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also
> >>>>>>>>>>>>>>>>>>>> gives
> >>>> us a
> >>>>>>>>>>>> point of
> >>>>>>>>>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer
> >>>>>>>>>>>>>>>>>>>> standing
> >>>>> issues
> >>>>>>>>>> that
> >>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider
> >>>>>>>>>>>>>>>>>>>> 1611,
> >>>>>>> 1525,
> >>>>>>>>>>>> 1258,
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and
> >>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>> beta
> >>>>>>>>>>>>>>>> release,
> >>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm
> >>>>>>>>>>>>>>>>>>>>> going
> >>>> to
> >>>>>>>>>>>> delay.
> >>>>>>>>>>>>>>>>> Here
> >>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in
> Airflow
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE &
> >>>>>>>>>>>>>>>>>>>>> NOTICE
> >>>>> issue
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>>>> SubDagOperator
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug
>  |airflow/jobs.py:create_dag_
> >>>>> run()
> >>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes
> can
> >>>> not
> >>>>>>> log
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
> >> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
> >>>> Repeated
> >>>>>>> if
> >>>>>>>>>>>>>>>> Email
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> Not
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >>>>> causes
> >>>>>>> it
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're
> >>>>>>>>>>>>>>>>>>>>> merged,
> >> I'll
> >>>>> cut
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be
> >>>>>>>>>>>>>>>>>>>>> really
> >>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini
> >>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>> 1.9.0?
> >>>>>>>>>> SLA
> >>>>>>>>>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its
> >>>>>>>>>>>>>>>>>>>>>>> not a
> >>>>>>> major
> >>>>>>>>>>>> bug,
> >>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
> >>>> deployment.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-98
> >>>>>>>>>>>>>>>>>>>>>>> 8 <
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2415
> >>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> >>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
> >>>>>>>>>>>>>>>>>>>>>>> __
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
> >>>>>>>>>>>>>>>>>>>>>>> __
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth,
> >>>>>>>>>>>>>>>>>>>>>>> TX
> >>>> 76102
> >>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |
> >>>>>>>>>>>>>>>>>>>>>>> www.simpli.fi <
> >>>>>>>>>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris
> >>>>>>>>>>>>>>>>>>>>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
> >>>> ryan.buckley@bluecore
> >>>>> .
> >>>>>>> com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken
> >>>>>>>>>>>>>>>>>>>>>>>>> on the
> >>>>>>> 1.9.0
> >>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>> incubator-airflow/pull/2590
> >>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong,
> >>>>>>>>>>>>>>>>>>>>>>>>> Fokko
> >>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>> release:
> >>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>> incubator-airflow/pull/2631
> >>>>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging
> >>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> >>>> is
> >>>>>>> not
> >>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>> (as
> >>>>>>>>>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> >>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Crawford
> >>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
> >> <mailto:
> >>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> type
> >> fix
> >>>>> in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>> jira/browse/AIRFLOW-1636
> >>>>> <
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>> jira/browse/AIRFLOW-1636
> >>>>> <
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>> incubator-airflow/pull/2626
> >>>>>>>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>> incubator-airflow/pull/2626
> >>>>>>>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
> >> blanked
> >>>>> out
> >>>>>>>>>> on
> >>>>>>>>>>>>>>>> edit
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >> criccomini@apache.org
> >>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
> >>>> looking
> >>>>>>>>>>>> forward
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guziel <
> >>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
> >>>> alex.guziel@airbnb.com
> >>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>> jira/browse/AIRFLOW-1519
> >>>>>>> <
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>>>> jira/browse/AIRFLOW-1621
> >>>>>>> <
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>> incubator-airflow/commit/
> >>>>>>>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>>>> incubator-airflow/commit/
> >>>>>>>>>> <
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>> good
> >>>>>>>>>> thing
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> >>>>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>> criccomini@apache.org>
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next
> >>>> week,
> >>>>>>>>>> then.
> >>>>>>>>>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
> >>>> can
> >>>>>>> get
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> de
> >>>>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a
> >>>>>>>>>>>> particular
> >>>>>>>>>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>> tarball
> >>>>>>>>>>>> than
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >> criccomini@apache.org
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
> >>>> significantly
> >>>>>>>>>>>>>>>> different?
> >>>>>>>>>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke de
> >>>>>>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>> <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>>> RCs
> >>>>>>>>>> right
> >>>>>>>>>>>>>>>> away?
> >>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
> >> Riccomini
> >>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >>>>> planning
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
> >> vote.
> >>>>>>> Once
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
> >>>> cherry
> >>>>>>>>>>>> picked
> >>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required
> >>>> to
> >>>>>>> get
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>> Riccomini
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>>>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here
> >> are
> >>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> >>>>> Vulnerability in
> >>>>>>>>>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> >>>>> logging in
> >>>>>>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
> >>>> of
> >>>>>>>>>> local
> >>>>>>>>>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> >>>>> logger to
> >>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> >>>>> LICENSE &
> >>>>>>>>>>>> NOTICE
> >>>>>>>>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> >>>>> duplicate
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
> >> to
> >>>>>>>>>> operate
> >>>>>>>>>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open
> |active_dagruns
> >>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler
> DAG
> >>>>>>>>>> processes
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> >>>>> displayed
> >>>>>>>>>> over
> >>>>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
> >>>>> running
> >>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor
> tests
> >>>>> and
> >>>>>>>>>> build
> >>>>>>>>>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
> >>>> should
> >>>>> not
> >>>>>>>>>>>> push
> >>>>>>>>>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
> >>>> size
> >>>>> for
> >>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> >>>>>>> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> >>>>> unexpectedly
> >>>>>>>>>>>>>>>> added to
> >>>>>>>>>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>>>>>>>>> cherry-picked
> >>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
> >>>> want
> >>>>>>> in,
> >>>>>>>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
> >>>> our
> >>>>>>> dev
> >>>>>>>>>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verifying
> >>>>>>> stability.
> >>>>>>>>>>>> If
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
> >>>> 1.9.0
> >>>>>>> test
> >>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
> --------------------------------------------------------------------
>
> Intel Technology Poland sp. z o.o.
> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII
> Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP
> 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>
> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego
> adresata i moze zawierac informacje poufne. W razie przypadkowego
> otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej
> usuniecie; jakiekolwiek
> przegladanie lub rozpowszechnianie jest zabronione.
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). If you are not the intended
> recipient, please contact the sender and delete all copies; any review or
> distribution by
> others is strictly prohibited.
>

RE: Airflow 1.9.0 status

Posted by "Cieplucha, Michal" <mi...@intel.com>.

Hello,

Will this: 
[AIRFLOW-1641] Handle executor events in the scheduler
https://github.com/apache/incubator-airflow/commit/2abead7049806482047e29d123a109b444c00355

be included in 1.9?

mC

-----Original Message-----
From: Bolke de Bruin [mailto:bdbruin@gmail.com] 
Sent: Wednesday, November 1, 2017 11:42 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow 1.9.0 status

I think we are ready for RC.

Bolke


> On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org> wrote:
> 
> Great, sounds like this is the last remaining JIRA, then?
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1764
> 
> On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> 
>> Already fixed.
>> 
>>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
>> wrote:
>>> 
>>> Is this bug issue on DAG scheduling
>>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
>> needs
>>> resolution before the RC?
>>> 
>>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini 
>>> <cr...@apache.org>
>>> wrote:
>>> 
>>>> Both blockers have been merged. Are we ready for an RC?
>>>> 
>>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin 
>>>> <bd...@gmail.com>
>>>> wrote:
>>>> 
>>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). 
>>>>> But
>> 1765
>>>>> is not a vulnerability, although *maybe* unexpected behaviour.
>>>>> 
>>>>> Bolke.
>>>>> 
>>>>> 
>>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>> I marked them as blocker. Still would like to understand how 
>>>>>> important
>>>>> this
>>>>>> is, though. The API is experimental, so I could see an argument 
>>>>>> for
>>>> them
>>>>>> not being blocker provided the vulnerability is limited to
>> experimental
>>>>>> areas.
>>>>>> 
>>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
>>>> ash@firemirror.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> I think we need to add
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>>>>>>> 
>>>>>>> to that list as critical security fixes. I'll tackled 1765 
>>>>>>> tomorrow
>>>> (UK
>>>>>>> time) if no one gets round to it before then.
>>>>>>> 
>>>>>>> -ash
>>>>>>> 
>>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini 
>>>>>>>> <cr...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> Current blockers for 1.9.0:
>>>>>>>> 
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
>>>>> criccomini@apache.org
>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
>> issues
>>>>> for
>>>>>>>>> 1.9.0:
>>>>>>>>> 
>>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
>> |task.retries
>>>>> can
>>>>>>>>> be False
>>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
>>>> custom
>>>>>>>>> config on PYTHONPATH
>>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
>>>>> stuck
>>>>>>>>> in queued state
>>>>>>>>> 
>>>>>>>>> PRs are out for them. After these three are merged, I'm 
>>>>>>>>> planning to
>>>>> cut
>>>>>>> an
>>>>>>>>> RC.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin 
>>>>>>>>> <bdbruin@gmail.com
>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I upgraded our production environment today. Some 
>>>>>>>>>> observations
>>>>>>>>>> 
>>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the 
>>>>>>>>>> box
>>>>>>> correctly
>>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to 
>>>>>>>>>> be
>>>> casted
>>>>>>> to an
>>>>>>>>>> int. Not sure how this gets to be false. This is a block imho 
>>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility 
>>>>>>>>>> sucks. We
>>>>>>> needed
>>>>>>>>>> to rework quite a lot of dags 3. We use LdAP for logins. We 
>>>>>>>>>> havent configured the ‘superuser’
>>>>>>>>>> ‘data_profiler’ groups and thus should have automatic 
>>>>>>>>>> superuser
>>>>>>> privileges.
>>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or 
>>>>>>>>>> see
>> the
>>>>>>> admin
>>>>>>>>>> screens).
>>>>>>>>>> 
>>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the 
>>>>>>>>>> new
>>>>>>> logging.
>>>>>>>>>> 
>>>>>>>>>> Bolke
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko 
>>>>>>>>>>> <fokko@driesprong.frl
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Bolke,
>>>>>>>>>>> 
>>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>>>>>>> /browse/AIRFLOW-1731.
>>>>>>>>>>> 
>>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is 
>>>>>>>>>>> on
>> the
>>>>>>> path:
>>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>>>>>>> 
>>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
>>>>>>> Hopefully
>>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with 
>>>>>>>>>>> the
>>>>> Spark
>>>>>>>>>>> Summit.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>> 
>>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>>>>>>>> 
>>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>>>>>>>> 
>>>>>>>>>>>> What are the issues that are open?
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
>> criccomini@apache.org
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week. 
>>>>>>>>>>>>> Something
>>>> for
>>>>>>> you
>>>>>>>>>>>> guys to play with.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
>>>>> bdbruin@gmail.com
>>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>> hi Chris,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied 
>>>>>>>>>>>>> with
>>>>>>> preparing
>>>>>>>>>>>> for a conference and the team is readying a release of one 
>>>>>>>>>>>> of
>> our
>>>>>>> core
>>>>>>>>>>>> products. So it will probably will be after this week when 
>>>>>>>>>>>> I get
>>>> my
>>>>>>>>>> hands
>>>>>>>>>>>> dirty again.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>>>>>>> criccomini@apache.org
>>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
>>>> Stable
>>>>>>>>>>>> releases
>>>>>>>>>>>>>> depend on community involvement.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's 
>>>>>>>>>>>>>>> not in
>>>>>>> alpha0,
>>>>>>>>>>>> but
>>>>>>>>>>>>>>> will be included in alpha1.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>>>>>>> <fenglu@google.com.invalid
>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in, 
>>>>>>>>>>>>>>>> but
>>>>> this
>>>>>>>>>> is a
>>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators, 
>>>>>>>>>>>>>>>> is
>> it
>>>>>>>>>>>> possible to
>>>>>>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
>> b3e985a3
>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3
>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
>>>> download
>>>>> it
>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airfl
>>>>>>>>>>>>>>>>> ow/1
>> .
>>>>>>>>>>>> 9.0alpha0/ 
>>>>>>>>>>>> <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
>>>> incubating-bin.tar.gz
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The goal is to have the community install and run this 
>>>>>>>>>>>>>>>>> to
>>>>> expose
>>>>>>>>>> any
>>>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>> SubDagOperator
>>>>>>> are
>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>> run()
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
>>>> log
>>>>>>> to
>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>> exception
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
>>>> it
>>>>>>> to
>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
>>>> :)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & 
>>>>>>>>>>>>>>>>>>> NOTICE
>>>> issue
>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>> SubDagOperator
>>>>>>>>>> are
>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>> causes
>>>>> it
>>>>>>>>>> to
>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch 
>>>>>>>>>>>>>>>>>>>> is
>> only
>>>>> cut
>>>>>>>>>> at
>>>>>>>>>>>> RC
>>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also 
>>>>>>>>>>>>>>>>>>>> gives
>>>> us a
>>>>>>>>>>>> point of
>>>>>>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer 
>>>>>>>>>>>>>>>>>>>> standing
>>>>> issues
>>>>>>>>>> that
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 
>>>>>>>>>>>>>>>>>>>> 1611,
>>>>>>> 1525,
>>>>>>>>>>>> 1258,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 
>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>> beta
>>>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm 
>>>>>>>>>>>>>>>>>>>>> going
>>>> to
>>>>>>>>>>>> delay.
>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & 
>>>>>>>>>>>>>>>>>>>>> NOTICE
>>>>> issue
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>> SubDagOperator
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>>>>> run()
>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
>>>> not
>>>>>>> log
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
>>>> Repeated
>>>>>>> if
>>>>>>>>>>>>>>>> Email
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>>> causes
>>>>>>> it
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're 
>>>>>>>>>>>>>>>>>>>>> merged,
>> I'll
>>>>> cut
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be 
>>>>>>>>>>>>>>>>>>>>> really
>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini 
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org 
>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>> 1.9.0?
>>>>>>>>>> SLA
>>>>>>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its 
>>>>>>>>>>>>>>>>>>>>>>> not a
>>>>>>> major
>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
>>>> deployment.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-98
>>>>>>>>>>>>>>>>>>>>>>> 8 <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/pull/2415
>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.* 
>>>>>>>>>>>>>>>>>>>>>>> ________________________________________________
>>>>>>>>>>>>>>>>>>>>>>> __
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, 
>>>>>>>>>>>>>>>>>>>>>>> TX
>>>> 76102
>>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  
>>>>>>>>>>>>>>>>>>>>>>> www.simpli.fi <
>>>>>>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris 
>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org 
>>>>>>>>>>>>>>>>>>>> <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
>>>> ryan.buckley@bluecore
>>>>> .
>>>>>>> com
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken 
>>>>>>>>>>>>>>>>>>>>>>>>> on the
>>>>>>> 1.9.0
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>> incubator-airflow/pull/2590
>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, 
>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/pull/2631
>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging 
>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>>>> is
>>>>>>> not
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Crawford
>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
>> <mailto:
>>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>> fix
>>>>> in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>> jira/browse/AIRFLOW-1636
>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>> jira/browse/AIRFLOW-1636
>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
>> blanked
>>>>> out
>>>>>>>>>> on
>>>>>>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org
>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
>>>> looking
>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guziel <
>>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
>>>> alex.guziel@airbnb.com
>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>> jira/browse/AIRFLOW-1519
>>>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>> jira/browse/AIRFLOW-1621
>>>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/commit/
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/commit/
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>> good
>>>>>>>>>> thing
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>> criccomini@apache.org>
>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next
>>>> week,
>>>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>> can
>>>>>>> get
>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> de
>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on a
>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>> tarball
>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Riccomini
>> <
>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org
>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
>>>> significantly
>>>>>>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke de
>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com 
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>> RCs
>>>>>>>>>> right
>>>>>>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
>> Riccomini
>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>> planning
>>>>>>>>>> to
>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
>> vote.
>>>>>>> Once
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>> cherry
>>>>>>>>>>>> picked
>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required
>>>> to
>>>>>>> get
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>> Riccomini
>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here
>> are
>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
>>>>> Vulnerability in
>>>>>>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
>>>>> logging in
>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source
>>>> of
>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
>>>>> logger to
>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
>>>>> LICENSE &
>>>>>>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
>>>>> duplicate
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
>> to
>>>>>>>>>> operate
>>>>>>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>>>>>>>>> processes
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
>>>>> displayed
>>>>>>>>>> over
>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
>>>>> running
>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
>>>>> and
>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
>>>> should
>>>>> not
>>>>>>>>>>>> push
>>>>>>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
>>>> size
>>>>> for
>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
>>>>> unexpectedly
>>>>>>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>>>>>>> cherry-picked
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
>>>> want
>>>>>>> in,
>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>> our
>>>>>>> dev
>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verifying
>>>>>>> stability.
>>>>>>>>>>>> If
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
>>>> 1.9.0
>>>>>>> test
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 


--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

I think we are ready for RC.

Bolke


> On 31 Oct 2017, at 21:30, Chris Riccomini <cr...@apache.org> wrote:
> 
> Great, sounds like this is the last remaining JIRA, then?
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1764
> 
> On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> 
>> Already fixed.
>> 
>>> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
>> wrote:
>>> 
>>> Is this bug issue on DAG scheduling
>>> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
>> needs
>>> resolution before the RC?
>>> 
>>> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>> 
>>>> Both blockers have been merged. Are we ready for an RC?
>>>> 
>>>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>> 
>>>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But
>> 1765
>>>>> is not a vulnerability, although *maybe* unexpected behaviour.
>>>>> 
>>>>> Bolke.
>>>>> 
>>>>> 
>>>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>> I marked them as blocker. Still would like to understand how important
>>>>> this
>>>>>> is, though. The API is experimental, so I could see an argument for
>>>> them
>>>>>> not being blocker provided the vulnerability is limited to
>> experimental
>>>>>> areas.
>>>>>> 
>>>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
>>>> ash@firemirror.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> I think we need to add
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>>>>>>> 
>>>>>>> to that list as critical security fixes. I'll tackled 1765 tomorrow
>>>> (UK
>>>>>>> time) if no one gets round to it before then.
>>>>>>> 
>>>>>>> -ash
>>>>>>> 
>>>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> Current blockers for 1.9.0:
>>>>>>>> 
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
>>>>> criccomini@apache.org
>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
>> issues
>>>>> for
>>>>>>>>> 1.9.0:
>>>>>>>>> 
>>>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
>> |task.retries
>>>>> can
>>>>>>>>> be False
>>>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
>>>> custom
>>>>>>>>> config on PYTHONPATH
>>>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
>>>>> stuck
>>>>>>>>> in queued state
>>>>>>>>> 
>>>>>>>>> PRs are out for them. After these three are merged, I'm planning to
>>>>> cut
>>>>>>> an
>>>>>>>>> RC.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bdbruin@gmail.com
>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I upgraded our production environment today. Some observations
>>>>>>>>>> 
>>>>>>>>>> 1. Database migration (add max tries) didn’t workout of the box
>>>>>>> correctly
>>>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to be
>>>> casted
>>>>>>> to an
>>>>>>>>>> int. Not sure how this gets to be false. This is a block imho
>>>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
>>>>>>> needed
>>>>>>>>>> to rework quite a lot of dags
>>>>>>>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>>>>>>>>>> ‘data_profiler’ groups and thus should have automatic superuser
>>>>>>> privileges.
>>>>>>>>>> We don’t at the moment (eg. we cannot manage connections or see
>> the
>>>>>>> admin
>>>>>>>>>> screens).
>>>>>>>>>> 
>>>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the new
>>>>>>> logging.
>>>>>>>>>> 
>>>>>>>>>> Bolke
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Bolke,
>>>>>>>>>>> 
>>>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>>>>>>> /browse/AIRFLOW-1731.
>>>>>>>>>>> 
>>>>>>>>>>> This requirers that we are sure that the /plugin/ folder is on
>> the
>>>>>>> path:
>>>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>>>>>>> 
>>>>>>>>>>> And maybe add an additional test and updating the docs a bit.
>>>>>>> Hopefully
>>>>>>>>>>> somewhere in the next few days, however I'm a bit busy with the
>>>>> Spark
>>>>>>>>>>> Summit.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>> 
>>>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>>>>>>>> 
>>>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>>>>>>>> 
>>>>>>>>>>>> What are the issues that are open?
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
>> criccomini@apache.org
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week. Something
>>>> for
>>>>>>> you
>>>>>>>>>>>> guys to play with.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
>>>>> bdbruin@gmail.com
>>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>> hi Chris,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied with
>>>>>>> preparing
>>>>>>>>>>>> for a conference and the team is readying a release of one of
>> our
>>>>>>> core
>>>>>>>>>>>> products. So it will probably will be after this week when I get
>>>> my
>>>>>>>>>> hands
>>>>>>>>>>>> dirty again.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>>>>>>> criccomini@apache.org
>>>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
>>>> Stable
>>>>>>>>>>>> releases
>>>>>>>>>>>>>> depend on community involvement.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
>>>>>>> alpha0,
>>>>>>>>>>>> but
>>>>>>>>>>>>>>> will be included in alpha1.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>>>>>>> <fenglu@google.com.invalid
>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in, but
>>>>> this
>>>>>>>>>> is a
>>>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators, is
>> it
>>>>>>>>>>>> possible to
>>>>>>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
>> b3e985a3
>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
>>>> download
>>>>> it
>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1
>> .
>>>>>>>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
>>>> incubating-bin.tar.gz
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The goal is to have the community install and run this to
>>>>> expose
>>>>>>>>>> any
>>>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>> SubDagOperator
>>>>>>> are
>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>> run()
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
>>>> log
>>>>>>> to
>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>> exception
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
>>>> it
>>>>>>> to
>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
>>>> :)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>>>> issue
>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>> SubDagOperator
>>>>>>>>>> are
>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>> causes
>>>>> it
>>>>>>>>>> to
>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is
>> only
>>>>> cut
>>>>>>>>>> at
>>>>>>>>>>>> RC
>>>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives
>>>> us a
>>>>>>>>>>>> point of
>>>>>>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer standing
>>>>> issues
>>>>>>>>>> that
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
>>>>>>> 1525,
>>>>>>>>>>>> 1258,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0
>>>> beta
>>>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going
>>>> to
>>>>>>>>>>>> delay.
>>>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>>>>> issue
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>>>> SubDagOperator
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>>>>> run()
>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
>>>> not
>>>>>>> log
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
>>>> Repeated
>>>>>>> if
>>>>>>>>>>>>>>>> Email
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>>>> causes
>>>>>>> it
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're merged,
>> I'll
>>>>> cut
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be really
>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
>>>>> 1.9.0?
>>>>>>>>>> SLA
>>>>>>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
>>>>>>> major
>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
>>>> deployment.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/pull/2415
>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX
>>>> 76102
>>>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>>>>>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
>>>> ryan.buckley@bluecore
>>>>> .
>>>>>>> com
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
>>>>>>> 1.9.0
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>> incubator-airflow/pull/2590
>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>>>>>>>>>> release:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/pull/2631
>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration
>>>> is
>>>>>>> not
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford
>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
>> <mailto:
>>>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type
>> fix
>>>>> in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>> jira/browse/AIRFLOW-1636
>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>> jira/browse/AIRFLOW-1636
>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/pull/2626
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
>> blanked
>>>>> out
>>>>>>>>>> on
>>>>>>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org
>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
>>>> looking
>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
>>>> alex.guziel@airbnb.com
>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>> jira/browse/AIRFLOW-1519
>>>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>> jira/browse/AIRFLOW-1621
>>>>>>> <
>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/commit/
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>>>> incubator-airflow/commit/
>>>>>>>>>> <
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a
>>>> good
>>>>>>>>>> thing
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
>>>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>> criccomini@apache.org>
>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next
>>>> week,
>>>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what
>>>> can
>>>>>>> get
>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
>>>>> tarball
>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini
>> <
>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org
>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
>>>> significantly
>>>>>>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
>>>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do
>>>> RCs
>>>>>>>>>> right
>>>>>>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
>> Riccomini
>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
>>>>> planning
>>>>>>>>>> to
>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
>> vote.
>>>>>>> Once
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get
>>>> cherry
>>>>>>>>>>>> picked
>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required
>>>> to
>>>>>>> get
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>>>>>>>>>> Riccomini
>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here
>> are
>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
>>>>> Vulnerability in
>>>>>>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
>>>>> logging in
>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source
>>>> of
>>>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
>>>>> logger to
>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
>>>>> LICENSE &
>>>>>>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
>>>>> duplicate
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
>> to
>>>>>>>>>> operate
>>>>>>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>>>>>>>>> processes
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
>>>>> displayed
>>>>>>>>>> over
>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
>>>>> running
>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
>>>>> and
>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
>>>> should
>>>>> not
>>>>>>>>>>>> push
>>>>>>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
>>>> size
>>>>> for
>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
>>>>> unexpectedly
>>>>>>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>>>>>>> cherry-picked
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you
>>>> want
>>>>>>> in,
>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into
>>>> our
>>>>>>> dev
>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
>>>>>>> stability.
>>>>>>>>>>>> If
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
>>>> 1.9.0
>>>>>>> test
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Great, sounds like this is the last remaining JIRA, then?

https://issues.apache.org/jira/browse/AIRFLOW-1764

On Tue, Oct 31, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> Already fixed.
>
> > On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com>
> wrote:
> >
> > Is this bug issue on DAG scheduling
> > <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that
> needs
> > resolution before the RC?
> >
> > On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >> Both blockers have been merged. Are we ready for an RC?
> >>
> >> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>
> >>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But
> 1765
> >>> is not a vulnerability, although *maybe* unexpected behaviour.
> >>>
> >>> Bolke.
> >>>
> >>>
> >>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
> >> wrote:
> >>>>
> >>>> I marked them as blocker. Still would like to understand how important
> >>> this
> >>>> is, though. The API is experimental, so I could see an argument for
> >> them
> >>>> not being blocker provided the vulnerability is limited to
> experimental
> >>>> areas.
> >>>>
> >>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
> >> ash@firemirror.com>
> >>>> wrote:
> >>>>
> >>>>> I think we need to add
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
> >>>>>
> >>>>> to that list as critical security fixes. I'll tackled 1765 tomorrow
> >> (UK
> >>>>> time) if no one gets round to it before then.
> >>>>>
> >>>>> -ash
> >>>>>
> >>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>> Hey all,
> >>>>>>
> >>>>>> Current blockers for 1.9.0:
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Chris
> >>>>>>
> >>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> >>> criccomini@apache.org
> >>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey all,
> >>>>>>>
> >>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking
> issues
> >>> for
> >>>>>>> 1.9.0:
> >>>>>>>
> >>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open
> |task.retries
> >>> can
> >>>>>>> be False
> >>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
> >> custom
> >>>>>>> config on PYTHONPATH
> >>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
> >>> stuck
> >>>>>>> in queued state
> >>>>>>>
> >>>>>>> PRs are out for them. After these three are merged, I'm planning to
> >>> cut
> >>>>> an
> >>>>>>> RC.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Chris
> >>>>>>>
> >>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bdbruin@gmail.com
> >
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> I upgraded our production environment today. Some observations
> >>>>>>>>
> >>>>>>>> 1. Database migration (add max tries) didn’t workout of the box
> >>>>> correctly
> >>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to be
> >> casted
> >>>>> to an
> >>>>>>>> int. Not sure how this gets to be false. This is a block imho
> >>>>>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
> >>>>> needed
> >>>>>>>> to rework quite a lot of dags
> >>>>>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
> >>>>>>>> ‘data_profiler’ groups and thus should have automatic superuser
> >>>>> privileges.
> >>>>>>>> We don’t at the moment (eg. we cannot manage connections or see
> the
> >>>>> admin
> >>>>>>>> screens).
> >>>>>>>>
> >>>>>>>> After fixing #1, the rest seems quite normal. Happy with the new
> >>>>> logging.
> >>>>>>>>
> >>>>>>>> Bolke
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Bolke,
> >>>>>>>>>
> >>>>>>>>> This one is a blocker: https://issues.apache.org/jira
> >>>>>>>> /browse/AIRFLOW-1731.
> >>>>>>>>>
> >>>>>>>>> This requirers that we are sure that the /plugin/ folder is on
> the
> >>>>> path:
> >>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>>>>>>
> >>>>>>>>> And maybe add an additional test and updating the docs a bit.
> >>>>> Hopefully
> >>>>>>>>> somewhere in the next few days, however I'm a bit busy with the
> >>> Spark
> >>>>>>>>> Summit.
> >>>>>>>>>
> >>>>>>>>> Cheers, Fokko
> >>>>>>>>>
> >>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>>>>>>>>>
> >>>>>>>>>> What are the issues that are open?
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>> Bolke
> >>>>>>>>>>
> >>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <
> criccomini@apache.org
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> K, I will plan on cutting an alpha1 later this week. Something
> >> for
> >>>>> you
> >>>>>>>>>> guys to play with.
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> >>> bdbruin@gmail.com
> >>>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>>> hi Chris,
> >>>>>>>>>>>
> >>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied with
> >>>>> preparing
> >>>>>>>>>> for a conference and the team is readying a release of one of
> our
> >>>>> core
> >>>>>>>>>> products. So it will probably will be after this week when I get
> >> my
> >>>>>>>> hands
> >>>>>>>>>> dirty again.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>
> >>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> >>>>> criccomini@apache.org
> >>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
> >> Stable
> >>>>>>>>>> releases
> >>>>>>>>>>>> depend on community involvement.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Chris
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
> >>>>> alpha0,
> >>>>>>>>>> but
> >>>>>>>>>>>>> will be included in alpha1.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> >>>>> <fenglu@google.com.invalid
> >>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I know it's annoying to have last minute commit com in, but
> >>> this
> >>>>>>>> is a
> >>>>>>>>>>>>>> highly desirable feature for folks using GCP operators, is
> it
> >>>>>>>>>> possible to
> >>>>>>>>>>>>>> include AIRFLOW-1635
> >>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/
> b3e985a3
> >> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>>>>>>> in?
> >>>>>>>>>>>>>> More than happy to offer help if there's something I can do.
> >>>>>>>>>>>>>> Thanks a lot.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
> >> download
> >>> it
> >>>>>>>>>> here:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1
> .
> >>>>>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>>>>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
> >> incubating-bin.tar.gz
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The goal is to have the community install and run this to
> >>> expose
> >>>>>>>> any
> >>>>>>>>>>>>>> bugs
> >>>>>>>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >> SubDagOperator
> >>>>> are
> >>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> run()
> >>>>>>>>>> exception
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
> >> log
> >>>>> to
> >>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>>>> exception
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
> >> it
> >>>>> to
> >>>>>>>>>> fail
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
> >> :)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> >> issue
> >>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>> SubDagOperator
> >>>>>>>> are
> >>>>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >> causes
> >>> it
> >>>>>>>> to
> >>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is
> only
> >>> cut
> >>>>>>>> at
> >>>>>>>>>> RC
> >>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives
> >> us a
> >>>>>>>>>> point of
> >>>>>>>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In addition the list below are mostly longer standing
> >>> issues
> >>>>>>>> that
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
> >>>>> 1525,
> >>>>>>>>>> 1258,
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0
> >> beta
> >>>>>>>>>>>>>> release,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going
> >> to
> >>>>>>>>>> delay.
> >>>>>>>>>>>>>>> Here
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> >>> issue
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >>>>> SubDagOperator
> >>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> >>> run()
> >>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
> >> not
> >>>>> log
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug
>  |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
> >> Repeated
> >>>>> if
> >>>>>>>>>>>>>> Email
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> Not
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> >>> causes
> >>>>> it
> >>>>>>>>>> to
> >>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're merged,
> I'll
> >>> cut
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be really
> >>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
> >>> 1.9.0?
> >>>>>>>> SLA
> >>>>>>>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
> >>>>> major
> >>>>>>>>>> bug,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
> >> deployment.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2415
> >> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> >>>>>>>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX
> >> 76102
> >>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> >>>>>>>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
> >> ryan.buckley@bluecore
> >>> .
> >>>>> com
> >>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
> >>>>> 1.9.0
> >>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2590
> >>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
> >>>>>>>> release:
> >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>> incubator-airflow/pull/2631
> >>>>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration
> >> is
> >>>>> not
> >>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>> (as
> >>>>>>>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> >>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
> <mailto:
> >>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type
> fix
> >>> in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >> jira/browse/AIRFLOW-1636
> >>> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >> jira/browse/AIRFLOW-1636
> >>> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>> incubator-airflow/pull/2626
> >>>>>>>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>> incubator-airflow/pull/2626
> >>>>>>>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting
> blanked
> >>> out
> >>>>>>>> on
> >>>>>>>>>>>>>> edit
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> criccomini@apache.org
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
> >> looking
> >>>>>>>>>> forward
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
> >> alex.guziel@airbnb.com
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>> jira/browse/AIRFLOW-1519
> >>>>> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> >>> jira/browse/AIRFLOW-1621
> >>>>> <
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>> incubator-airflow/commit/
> >>>>>>>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >>>>> incubator-airflow/commit/
> >>>>>>>> <
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a
> >> good
> >>>>>>>> thing
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> >>> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>> criccomini@apache.org>
> >>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next
> >> week,
> >>>>>>>> then.
> >>>>>>>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what
> >> can
> >>>>> get
> >>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
> >>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> >>>>>>>>>> particular
> >>>>>>>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
> >>> tarball
> >>>>>>>>>> than
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini
> <
> >>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> criccomini@apache.org
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
> >> significantly
> >>>>>>>>>>>>>> different?
> >>>>>>>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
> >>>>> Bruin <
> >>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do
> >> RCs
> >>>>>>>> right
> >>>>>>>>>>>>>> away?
> >>>>>>>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
> Riccomini
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
> >>> planning
> >>>>>>>> to
> >>>>>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release
> vote.
> >>>>> Once
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get
> >> cherry
> >>>>>>>>>> picked
> >>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required
> >> to
> >>>>> get
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
> >>>>>>>> Riccomini
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here
> are
> >>> the
> >>>>>>>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> >>> Vulnerability in
> >>>>>>>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> >>> logging in
> >>>>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source
> >> of
> >>>>>>>> local
> >>>>>>>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> >>> logger to
> >>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> >>> LICENSE &
> >>>>>>>>>> NOTICE
> >>>>>>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> >>> duplicate
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
> to
> >>>>>>>> operate
> >>>>>>>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> >>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
> >>>>>>>> processes
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> >>> displayed
> >>>>>>>> over
> >>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
> >>> running
> >>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
> >>> and
> >>>>>>>> build
> >>>>>>>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
> >> should
> >>> not
> >>>>>>>>>> push
> >>>>>>>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
> >> size
> >>> for
> >>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> >>>>> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> >>> unexpectedly
> >>>>>>>>>>>>>> added to
> >>>>>>>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>>>>>>> cherry-picked
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you
> >> want
> >>>>> in,
> >>>>>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into
> >> our
> >>>>> dev
> >>>>>>>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
> >>>>> stability.
> >>>>>>>>>> If
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
> >> 1.9.0
> >>>>> test
> >>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

Already fixed.

> On 31 Oct 2017, at 18:49, Andy Hadjigeorgiou <an...@gmail.com> wrote:
> 
> Is this bug issue on DAG scheduling
> <https://issues.apache.org/jira/browse/AIRFLOW-1767> something that needs
> resolution before the RC?
> 
> On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini <cr...@apache.org>
> wrote:
> 
>> Both blockers have been merged. Are we ready for an RC?
>> 
>> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>> 
>>> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But 1765
>>> is not a vulnerability, although *maybe* unexpected behaviour.
>>> 
>>> Bolke.
>>> 
>>> 
>>>> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
>> wrote:
>>>> 
>>>> I marked them as blocker. Still would like to understand how important
>>> this
>>>> is, though. The API is experimental, so I could see an argument for
>> them
>>>> not being blocker provided the vulnerability is limited to experimental
>>>> areas.
>>>> 
>>>> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
>> ash@firemirror.com>
>>>> wrote:
>>>> 
>>>>> I think we need to add
>>>>> 
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>>>>> 
>>>>> to that list as critical security fixes. I'll tackled 1765 tomorrow
>> (UK
>>>>> time) if no one gets round to it before then.
>>>>> 
>>>>> -ash
>>>>> 
>>>>>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>>>>> 
>>>>>> Hey all,
>>>>>> 
>>>>>> Current blockers for 1.9.0:
>>>>>> 
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>> 
>>>>>> Cheers,
>>>>>> Chris
>>>>>> 
>>>>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
>>> criccomini@apache.org
>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> Talked with @Bolke on Gitter. Here are the currently blocking issues
>>> for
>>>>>>> 1.9.0:
>>>>>>> 
>>>>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries
>>> can
>>>>>>> be False
>>>>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
>> custom
>>>>>>> config on PYTHONPATH
>>>>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
>>> stuck
>>>>>>> in queued state
>>>>>>> 
>>>>>>> PRs are out for them. After these three are merged, I'm planning to
>>> cut
>>>>> an
>>>>>>> RC.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> I upgraded our production environment today. Some observations
>>>>>>>> 
>>>>>>>> 1. Database migration (add max tries) didn’t workout of the box
>>>>> correctly
>>>>>>>> for Postgres. “max_tries = False” didn’t work and needed to be
>> casted
>>>>> to an
>>>>>>>> int. Not sure how this gets to be false. This is a block imho
>>>>>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
>>>>> needed
>>>>>>>> to rework quite a lot of dags
>>>>>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>>>>>>>> ‘data_profiler’ groups and thus should have automatic superuser
>>>>> privileges.
>>>>>>>> We don’t at the moment (eg. we cannot manage connections or see the
>>>>> admin
>>>>>>>> screens).
>>>>>>>> 
>>>>>>>> After fixing #1, the rest seems quite normal. Happy with the new
>>>>> logging.
>>>>>>>> 
>>>>>>>> Bolke
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl
>>> 
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Bolke,
>>>>>>>>> 
>>>>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>>>>> /browse/AIRFLOW-1731.
>>>>>>>>> 
>>>>>>>>> This requirers that we are sure that the /plugin/ folder is on the
>>>>> path:
>>>>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>>>>> 
>>>>>>>>> And maybe add an additional test and updating the docs a bit.
>>>>> Hopefully
>>>>>>>>> somewhere in the next few days, however I'm a bit busy with the
>>> Spark
>>>>>>>>> Summit.
>>>>>>>>> 
>>>>>>>>> Cheers, Fokko
>>>>>>>>> 
>>>>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>>>>>> 
>>>>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>>>>>> 
>>>>>>>>>> What are the issues that are open?
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> Bolke
>>>>>>>>>> 
>>>>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <criccomini@apache.org
>>> 
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> K, I will plan on cutting an alpha1 later this week. Something
>> for
>>>>> you
>>>>>>>>>> guys to play with.
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
>>> bdbruin@gmail.com
>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>> hi Chris,
>>>>>>>>>>> 
>>>>>>>>>>> We are still running pre alpha. I am a bit preoccupied with
>>>>> preparing
>>>>>>>>>> for a conference and the team is readying a release of one of our
>>>>> core
>>>>>>>>>> products. So it will probably will be after this week when I get
>> my
>>>>>>>> hands
>>>>>>>>>> dirty again.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers
>>>>>>>>>>> Bolke
>>>>>>>>>>> 
>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>> 
>>>>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>>>>> criccomini@apache.org
>>>>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Checking in. Has anyone tested this in their environments?
>> Stable
>>>>>>>>>> releases
>>>>>>>>>>>> depend on community involvement.
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
>>>>> alpha0,
>>>>>>>>>> but
>>>>>>>>>>>>> will be included in alpha1.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>>>>> <fenglu@google.com.invalid
>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I know it's annoying to have last minute commit com in, but
>>> this
>>>>>>>> is a
>>>>>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
>>>>>>>>>> possible to
>>>>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3
>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
>> download
>>> it
>>>>>>>>>> here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>>>>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
>> incubating-bin.tar.gz
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The goal is to have the community install and run this to
>>> expose
>>>>>>>> any
>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>> SubDagOperator
>>>>> are
>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>>>>> exception
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
>> log
>>>>> to
>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>> exception
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
>> it
>>>>> to
>>>>>>>>>> fail
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
>> :)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>> issue
>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>> SubDagOperator
>>>>>>>> are
>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>> causes
>>> it
>>>>>>>> to
>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only
>>> cut
>>>>>>>> at
>>>>>>>>>> RC
>>>>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives
>> us a
>>>>>>>>>> point of
>>>>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> In addition the list below are mostly longer standing
>>> issues
>>>>>>>> that
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
>>>>> 1525,
>>>>>>>>>> 1258,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0
>> beta
>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going
>> to
>>>>>>>>>> delay.
>>>>>>>>>>>>>>> Here
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
>>> issue
>>>>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>>>>> SubDagOperator
>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
>>> run()
>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
>> not
>>>>> log
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
>> Repeated
>>>>> if
>>>>>>>>>>>>>> Email
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
>>> causes
>>>>> it
>>>>>>>>>> to
>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll
>>> cut
>>>>>>>> the
>>>>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If you can help clean this up, that would be really
>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
>>> 1.9.0?
>>>>>>>> SLA
>>>>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
>>>>> major
>>>>>>>>>> bug,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
>> deployment.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415
>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX
>> 76102
>>>>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>>>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
>> ryan.buckley@bluecore
>>> .
>>>>> com
>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
>>>>> 1.9.0
>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/pull/2590
>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>>>>>>>> release:
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>> incubator-airflow/pull/2631
>>>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration
>> is
>>>>> not
>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford
>> <
>>>>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>>>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix
>>> in?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>> jira/browse/AIRFLOW-1636
>>> <
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>> jira/browse/AIRFLOW-1636
>>> <
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/pull/2626
>>>>>>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/pull/2626
>>>>>>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked
>>> out
>>>>>>>> on
>>>>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
>> looking
>>>>>>>>>> forward
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
>> alex.guziel@airbnb.com
>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>> jira/browse/AIRFLOW-1519
>>>>> <
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
>>> jira/browse/AIRFLOW-1621
>>>>> <
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/commit/
>>>>>>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>>>>> incubator-airflow/commit/
>>>>>>>> <
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a
>> good
>>>>>>>> thing
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
>>> Riccomini <
>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>> criccomini@apache.org>
>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next
>> week,
>>>>>>>> then.
>>>>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what
>> can
>>>>> get
>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
>>> tarball
>>>>>>>>>> than
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org
>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
>> significantly
>>>>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
>>>>> Bruin <
>>>>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do
>> RCs
>>>>>>>> right
>>>>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini
>> <
>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
>>> planning
>>>>>>>> to
>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
>>>>> Once
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get
>> cherry
>>>>>>>>>> picked
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required
>> to
>>>>> get
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>>>>>>>> Riccomini
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are
>>> the
>>>>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
>>> Vulnerability in
>>>>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
>>> logging in
>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source
>> of
>>>>>>>> local
>>>>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
>>> logger to
>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
>>> LICENSE &
>>>>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
>>> duplicate
>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
>>>>>>>> operate
>>>>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>>>>>>> processes
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
>>> displayed
>>>>>>>> over
>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
>>> running
>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
>>> and
>>>>>>>> build
>>>>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
>> should
>>> not
>>>>>>>>>> push
>>>>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
>> size
>>> for
>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
>>> unexpectedly
>>>>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>>>>> cherry-picked
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you
>> want
>>>>> in,
>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into
>> our
>>>>> dev
>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
>>>>> stability.
>>>>>>>>>> If
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
>> 1.9.0
>>>>> test
>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: Airflow 1.9.0 status

Posted by Andy Hadjigeorgiou <an...@gmail.com>.

Is this bug issue on DAG scheduling
<https://issues.apache.org/jira/browse/AIRFLOW-1767> something that needs
resolution before the RC?

On Tue, Oct 31, 2017 at 1:17 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Both blockers have been merged. Are we ready for an RC?
>
> On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
>
> > AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But 1765
> > is not a vulnerability, although *maybe* unexpected behaviour.
> >
> > Bolke.
> >
> >
> > > On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org>
> wrote:
> > >
> > > I marked them as blocker. Still would like to understand how important
> > this
> > > is, though. The API is experimental, so I could see an argument for
> them
> > > not being blocker provided the vulnerability is limited to experimental
> > > areas.
> > >
> > > On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <
> ash@firemirror.com>
> > > wrote:
> > >
> > >> I think we need to add
> > >>
> > >> https://issues.apache.org/jira/browse/AIRFLOW-1764
> > >> https://issues.apache.org/jira/browse/AIRFLOW-1765
> > >>
> > >> to that list as critical security fixes. I'll tackled 1765 tomorrow
> (UK
> > >> time) if no one gets round to it before then.
> > >>
> > >> -ash
> > >>
> > >>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org>
> > wrote:
> > >>>
> > >>> Hey all,
> > >>>
> > >>> Current blockers for 1.9.0:
> > >>>
> > >>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> > >>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> > >>>
> > >>> Cheers,
> > >>> Chris
> > >>>
> > >>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> > criccomini@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hey all,
> > >>>>
> > >>>> Talked with @Bolke on Gitter. Here are the currently blocking issues
> > for
> > >>>> 1.9.0:
> > >>>>
> > >>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries
> > can
> > >>>> be False
> > >>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import
> custom
> > >>>> config on PYTHONPATH
> > >>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
> > stuck
> > >>>> in queued state
> > >>>>
> > >>>> PRs are out for them. After these three are merged, I'm planning to
> > cut
> > >> an
> > >>>> RC.
> > >>>>
> > >>>> Cheers,
> > >>>> Chris
> > >>>>
> > >>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> I upgraded our production environment today. Some observations
> > >>>>>
> > >>>>> 1. Database migration (add max tries) didn’t workout of the box
> > >> correctly
> > >>>>> for Postgres. “max_tries = False” didn’t work and needed to be
> casted
> > >> to an
> > >>>>> int. Not sure how this gets to be false. This is a block imho
> > >>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
> > >> needed
> > >>>>> to rework quite a lot of dags
> > >>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
> > >>>>> ‘data_profiler’ groups and thus should have automatic superuser
> > >> privileges.
> > >>>>> We don’t at the moment (eg. we cannot manage connections or see the
> > >> admin
> > >>>>> screens).
> > >>>>>
> > >>>>> After fixing #1, the rest seems quite normal. Happy with the new
> > >> logging.
> > >>>>>
> > >>>>> Bolke
> > >>>>>
> > >>>>>
> > >>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl
> >
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> Hi Bolke,
> > >>>>>>
> > >>>>>> This one is a blocker: https://issues.apache.org/jira
> > >>>>> /browse/AIRFLOW-1731.
> > >>>>>>
> > >>>>>> This requirers that we are sure that the /plugin/ folder is on the
> > >> path:
> > >>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> > >>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> > >>>>>>
> > >>>>>> And maybe add an additional test and updating the docs a bit.
> > >> Hopefully
> > >>>>>> somewhere in the next few days, however I'm a bit busy with the
> > Spark
> > >>>>>> Summit.
> > >>>>>>
> > >>>>>> Cheers, Fokko
> > >>>>>>
> > >>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> > >>>>>>
> > >>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> > >>>>>>>
> > >>>>>>> What are the issues that are open?
> > >>>>>>>
> > >>>>>>> Cheers
> > >>>>>>> Bolke
> > >>>>>>>
> > >>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <criccomini@apache.org
> >
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> K, I will plan on cutting an alpha1 later this week. Something
> for
> > >> you
> > >>>>>>> guys to play with.
> > >>>>>>>>
> > >>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> > bdbruin@gmail.com
> > >>>>>>> <ma...@gmail.com>> wrote:
> > >>>>>>>> hi Chris,
> > >>>>>>>>
> > >>>>>>>> We are still running pre alpha. I am a bit preoccupied with
> > >> preparing
> > >>>>>>> for a conference and the team is readying a release of one of our
> > >> core
> > >>>>>>> products. So it will probably will be after this week when I get
> my
> > >>>>> hands
> > >>>>>>> dirty again.
> > >>>>>>>>
> > >>>>>>>> Cheers
> > >>>>>>>> Bolke
> > >>>>>>>>
> > >>>>>>>> Verstuurd vanaf mijn iPad
> > >>>>>>>>
> > >>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> > >> criccomini@apache.org
> > >>>>>>> <ma...@apache.org>> het volgende geschreven:
> > >>>>>>>>>
> > >>>>>>>>> Hey all,
> > >>>>>>>>>
> > >>>>>>>>> Checking in. Has anyone tested this in their environments?
> Stable
> > >>>>>>> releases
> > >>>>>>>>> depend on community involvement.
> > >>>>>>>>>
> > >>>>>>>>> Cheers,
> > >>>>>>>>> Chris
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> > >>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
> > >> alpha0,
> > >>>>>>> but
> > >>>>>>>>>> will be included in alpha1.
> > >>>>>>>>>>
> > >>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> > >> <fenglu@google.com.invalid
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Chris,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I know it's annoying to have last minute commit com in, but
> > this
> > >>>>> is a
> > >>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
> > >>>>>>> possible to
> > >>>>>>>>>>> include AIRFLOW-1635
> > >>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3
> <
> > >>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> > >>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> > >>>>>>>>>>> in?
> > >>>>>>>>>>> More than happy to offer help if there's something I can do.
> > >>>>>>>>>>> Thanks a lot.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Feng
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> > >>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can
> download
> > it
> > >>>>>>> here:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> > >>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> > >>>>>>> airflow/1.9.0alpha0/>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The bin tarball can be installed with pip:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+
> incubating-bin.tar.gz
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The goal is to have the community install and run this to
> > expose
> > >>>>> any
> > >>>>>>>>>>> bugs
> > >>>>>>>>>>>> before we move on to official release candidates.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> SubDagOperator
> > >> are
> > >>>>>>>>>>> marked
> > >>>>>>>>>>>> as
> > >>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> > >>>>>>> exception
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>> @on
> > >>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
> log
> > >> to
> > >>>>>>>>>>> stdout
> > >>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> > >>>>> exception
> > >>>>>>>>>>> for
> > >>>>>>>>>>>> @once
> > >>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
> it
> > >> to
> > >>>>>>> fail
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>> Chris
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> > >>>>>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
> :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> > >>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Blockers for 1.9.0 are:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> issue
> > >>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> > SubDagOperator
> > >>>>> are
> > >>>>>>>>>>>> marked
> > >>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> causes
> > it
> > >>>>> to
> > >>>>>>>>>>> fail
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> > >>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Chris
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only
> > cut
> > >>>>> at
> > >>>>>>> RC
> > >>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives
> us a
> > >>>>>>> point of
> > >>>>>>>>>>>>>>> reference.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In addition the list below are mostly longer standing
> > issues
> > >>>>> that
> > >>>>>>>>>>> are
> > >>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
> > >> 1525,
> > >>>>>>> 1258,
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> 976 as blocker?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Cheers
> > >>>>>>>>>>>>>>> Bolke
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> > >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>>>> het volgende geschreven:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0
> beta
> > >>>>>>>>>>> release,
> > >>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going
> to
> > >>>>>>> delay.
> > >>>>>>>>>>>> Here
> > >>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>> the bugs that I'm tracking:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> > issue
> > >>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> > >> SubDagOperator
> > >>>>>>> are
> > >>>>>>>>>>>>>>> marked as
> > >>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> > run()
> > >>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> @on
> > >>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can
> not
> > >> log
> > >>>>>>> to
> > >>>>>>>>>>>>>>> stdout
> > >>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> > >>>>>>>>>>> exception
> > >>>>>>>>>>>>>>> for @once
> > >>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are
> Repeated
> > >> if
> > >>>>>>>>>>> Email
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> Not
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> > causes
> > >> it
> > >>>>>>> to
> > >>>>>>>>>>>> fail
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll
> > cut
> > >>>>> the
> > >>>>>>>>>>>>>>>> v1-9-stable and beta release.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> If you can help clean this up, that would be really
> > >>>>> appreciated.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> > >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> > >>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
> > 1.9.0?
> > >>>>> SLA
> > >>>>>>>>>>>>>>> callbacks
> > >>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
> > >> major
> > >>>>>>> bug,
> > >>>>>>>>>>>> but
> > >>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> does cause us some annoyance in our current
> deployment.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Link to Jira:
> > >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Link to PR:
> > >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415
> <
> > >>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>>> Charlie Jones
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> CHARLIE JONES
> > >>>>>>>>>>>>>>>>>> Data Engineer
> > >>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> > >>>>>>> 972.821.7631 <tel:972.821.7631>
> > >>>>>>>>>>>>>>>>>> __________________________________________________
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> > >>>>>>>>>>>>>>>>>> __________________________________________________
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX
> 76102
> > >>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> > >>>>>>> http://www.simpli.fi/>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> > >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Merged.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> > >>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:
> ryan.buckley@bluecore
> > .
> > >> com
> > >>>>>>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> > >>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
> > >> 1.9.0
> > >>>>>>>>>>> branch
> > >>>>>>>>>>>>>>> due
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> this issue.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2590
> > <
> > >>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>> Ryan
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> > >>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
> > >>>>> release:
> > >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> > incubator-airflow/pull/2631
> > >> <
> > >>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration
> is
> > >> not
> > >>>>>>>>>>> work
> > >>>>>>>>>>>> (as
> > >>>>>>>>>>>>>>>>>> far
> > >>>>>>>>>>>>>>>>>>>> as I
> > >>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> > >> appreciated.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> > >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Done!
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford
> <
> > >>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> > >>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix
> > in?
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1636
> > <
> > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> > >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1636
> > <
> > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> > >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> > >> incubator-airflow/pull/2626
> > >>>>> <
> > >>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> > >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> > >> incubator-airflow/pull/2626
> > >>>>> <
> > >>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked
> > out
> > >>>>> on
> > >>>>>>>>>>> edit
> > >>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>>>>>>> types.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>> Mike
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org
> >>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been
> looking
> > >>>>>>> forward
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>>>>>> fixes!
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > >>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:
> alex.guziel@airbnb.com
> > >
> > >>>>>>>>>>>>>>>>>>>>> .
> > >>>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> > jira/browse/AIRFLOW-1519
> > >> <
> > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> > >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> > jira/browse/AIRFLOW-1621
> > >> <
> > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> > >> incubator-airflow/commit/
> > >>>>> <
> > >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> > >>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> > >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> > >> incubator-airflow/commit/
> > >>>>> <
> > >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a
> good
> > >>>>> thing
> > >>>>>>> to
> > >>>>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> 1.9.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> > Riccomini <
> > >>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> > criccomini@apache.org>
> > >>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next
> week,
> > >>>>> then.
> > >>>>>>>>>>>>>>>>>> Initial
> > >>>>>>>>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what
> can
> > >> get
> > >>>>>>> into
> > >>>>>>>>>>>>>>>>>> 1.9.0
> > >>>>>>>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>> point.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
> > Bruin <
> > >>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> > >>>>>>> particular
> > >>>>>>>>>>>>>>>>>> fixed
> > >>>>>>>>>>>>>>>>>>>>> point
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
> > tarball
> > >>>>>>> than
> > >>>>>>>>>>> to
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>> git
> > >>>>>>>>>>>>>>>>>>>>>> pull.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org
> >
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process
> significantly
> > >>>>>>>>>>> different?
> > >>>>>>>>>>>>>>>>>>> IIRC,
> > >>>>>>>>>>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
> > >> Bruin <
> > >>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do
> RCs
> > >>>>> right
> > >>>>>>>>>>> away?
> > >>>>>>>>>>>>>>>>>>>> Isn’t a
> > >>>>>>>>>>>>>>>>>>>>>>>>> beta
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini
> <
> > >>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> criccomini@apache.org>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
> > planning
> > >>>>> to
> > >>>>>>>>>>> cut
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> stable
> > >>>>>>>>>>>>>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
> > >> Once
> > >>>>>>> the
> > >>>>>>>>>>>>>>>>>> stable
> > >>>>>>>>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get
> cherry
> > >>>>>>> picked
> > >>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> branch,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required
> to
> > >> get
> > >>>>>>> the
> > >>>>>>>>>>>>>>>>>> release
> > >>>>>>>>>>>>>>>>>>>>> out.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
> > >>>>> Riccomini
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> > >>>>> criccomini@apache.org>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are
> > the
> > >>>>>>>>>>>>>>>>>> outstanding
> > >>>>>>>>>>>>>>>>>>> PRs
> > >>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> > Vulnerability in
> > >>>>>>>>>>> Variable
> > >>>>>>>>>>>>>>>>>>>>> endpoint
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> > logging in
> > >>>>>>>>>>> Airflow
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source
> of
> > >>>>> local
> > >>>>>>>>>>>>>>>>>> loggers
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> > logger to
> > >>>>> log
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> > LICENSE &
> > >>>>>>> NOTICE
> > >>>>>>>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> > duplicate
> > >>>>> and
> > >>>>>>>>>>>>>>>>>> unneeded
> > >>>>>>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
> > >>>>> operate
> > >>>>>>>>>>> HDFS
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> > >>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> > >>>>>>>>>>>>>>>>>> n()
> > >>>>>>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> > >>>>> shouldn't
> > >>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>> paused
> > >>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
> > >>>>> processes
> > >>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>> log
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> > displayed
> > >>>>> over
> > >>>>>>>>>>> task
> > >>>>>>>>>>>>>>>>>>>>> instances
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> > >>>>>>>>>>> |airflow/jobs.py:manage_slas()
> > >>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
> > running
> > >>>>>>> task
> > >>>>>>>>>>>>>>>>>> causes
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> fail
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> > >>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> > >>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> > >>>>>>>>>>>>>>>>>>>>>>>>> job
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
> > and
> > >>>>> build
> > >>>>>>>>>>>>>>>>>> matrix
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
> should
> > not
> > >>>>>>> push
> > >>>>>>>>>>>>>>>>>> XComs
> > >>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum
> size
> > for
> > >>>>>>> XComs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> > >> semantics
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> > unexpectedly
> > >>>>>>>>>>> added to
> > >>>>>>>>>>>>>>>>>>> hive
> > >>>>>>>>>>>>>>>>>>>>> conf
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> > >>>>> cherry-picked
> > >>>>>>>>>>> into
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> 1.9.0
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you
> want
> > >> in,
> > >>>>>>>>>>> please
> > >>>>>>>>>>>>>>>>>> set
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> fix
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into
> our
> > >> dev
> > >>>>>>>>>>>> cluster,
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
> > >> stability.
> > >>>>>>> If
> > >>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>> run
> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the
> 1.9.0
> > >> test
> > >>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>> somewhere,
> > >>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Both blockers have been merged. Are we ready for an RC?

On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But 1765
> is not a vulnerability, although *maybe* unexpected behaviour.
>
> Bolke.
>
>
> > On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org> wrote:
> >
> > I marked them as blocker. Still would like to understand how important
> this
> > is, though. The API is experimental, so I could see an argument for them
> > not being blocker provided the vulnerability is limited to experimental
> > areas.
> >
> > On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <as...@firemirror.com>
> > wrote:
> >
> >> I think we need to add
> >>
> >> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >> https://issues.apache.org/jira/browse/AIRFLOW-1765
> >>
> >> to that list as critical security fixes. I'll tackled 1765 tomorrow (UK
> >> time) if no one gets round to it before then.
> >>
> >> -ash
> >>
> >>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org>
> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> Current blockers for 1.9.0:
> >>>
> >>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> >>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> criccomini@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> Talked with @Bolke on Gitter. Here are the currently blocking issues
> for
> >>>> 1.9.0:
> >>>>
> >>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries
> can
> >>>> be False
> >>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
> >>>> config on PYTHONPATH
> >>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
> stuck
> >>>> in queued state
> >>>>
> >>>> PRs are out for them. After these three are merged, I'm planning to
> cut
> >> an
> >>>> RC.
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>>>
> >>>>> I upgraded our production environment today. Some observations
> >>>>>
> >>>>> 1. Database migration (add max tries) didn’t workout of the box
> >> correctly
> >>>>> for Postgres. “max_tries = False” didn’t work and needed to be casted
> >> to an
> >>>>> int. Not sure how this gets to be false. This is a block imho
> >>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
> >> needed
> >>>>> to rework quite a lot of dags
> >>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
> >>>>> ‘data_profiler’ groups and thus should have automatic superuser
> >> privileges.
> >>>>> We don’t at the moment (eg. we cannot manage connections or see the
> >> admin
> >>>>> screens).
> >>>>>
> >>>>> After fixing #1, the rest seems quite normal. Happy with the new
> >> logging.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>>
> >>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Bolke,
> >>>>>>
> >>>>>> This one is a blocker: https://issues.apache.org/jira
> >>>>> /browse/AIRFLOW-1731.
> >>>>>>
> >>>>>> This requirers that we are sure that the /plugin/ folder is on the
> >> path:
> >>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>>>
> >>>>>> And maybe add an additional test and updating the docs a bit.
> >> Hopefully
> >>>>>> somewhere in the next few days, however I'm a bit busy with the
> Spark
> >>>>>> Summit.
> >>>>>>
> >>>>>> Cheers, Fokko
> >>>>>>
> >>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> >>>>>>
> >>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>>>>>>
> >>>>>>> What are the issues that are open?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Bolke
> >>>>>>>
> >>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> K, I will plan on cutting an alpha1 later this week. Something for
> >> you
> >>>>>>> guys to play with.
> >>>>>>>>
> >>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> bdbruin@gmail.com
> >>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>> hi Chris,
> >>>>>>>>
> >>>>>>>> We are still running pre alpha. I am a bit preoccupied with
> >> preparing
> >>>>>>> for a conference and the team is readying a release of one of our
> >> core
> >>>>>>> products. So it will probably will be after this week when I get my
> >>>>> hands
> >>>>>>> dirty again.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Bolke
> >>>>>>>>
> >>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>
> >>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> >> criccomini@apache.org
> >>>>>>> <ma...@apache.org>> het volgende geschreven:
> >>>>>>>>>
> >>>>>>>>> Hey all,
> >>>>>>>>>
> >>>>>>>>> Checking in. Has anyone tested this in their environments? Stable
> >>>>>>> releases
> >>>>>>>>> depend on community involvement.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
> >> alpha0,
> >>>>>>> but
> >>>>>>>>>> will be included in alpha1.
> >>>>>>>>>>
> >>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> >> <fenglu@google.com.invalid
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>
> >>>>>>>>>>> I know it's annoying to have last minute commit com in, but
> this
> >>>>> is a
> >>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
> >>>>>>> possible to
> >>>>>>>>>>> include AIRFLOW-1635
> >>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>>>> in?
> >>>>>>>>>>> More than happy to offer help if there's something I can do.
> >>>>>>>>>>> Thanks a lot.
> >>>>>>>>>>>
> >>>>>>>>>>> Feng
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download
> it
> >>>>>>> here:
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> >>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>>>
> >>>>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>>>
> >>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>>>>>>>>>
> >>>>>>>>>>>> The goal is to have the community install and run this to
> expose
> >>>>> any
> >>>>>>>>>>> bugs
> >>>>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>>>
> >>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> >> are
> >>>>>>>>>>> marked
> >>>>>>>>>>>> as
> >>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>>>>> exception
> >>>>>>>>>>>> for
> >>>>>>>>>>>> @on
> >>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
> >> to
> >>>>>>>>>>> stdout
> >>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>> exception
> >>>>>>>>>>> for
> >>>>>>>>>>>> @once
> >>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> >> to
> >>>>>>> fail
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Chris
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> SubDagOperator
> >>>>> are
> >>>>>>>>>>>> marked
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
> it
> >>>>> to
> >>>>>>>>>>> fail
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only
> cut
> >>>>> at
> >>>>>>> RC
> >>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
> >>>>>>> point of
> >>>>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In addition the list below are mostly longer standing
> issues
> >>>>> that
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
> >> 1525,
> >>>>>>> 1258,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>>>>>>>>>> release,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
> >>>>>>> delay.
> >>>>>>>>>>>> Here
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> issue
> >>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >> SubDagOperator
> >>>>>>> are
> >>>>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> run()
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
> >> log
> >>>>>>> to
> >>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>>>>>>> exception
> >>>>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated
> >> if
> >>>>>>>>>>> Email
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> Not
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> causes
> >> it
> >>>>>>> to
> >>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll
> cut
> >>>>> the
> >>>>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you can help clean this up, that would be really
> >>>>> appreciated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
> 1.9.0?
> >>>>> SLA
> >>>>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
> >> major
> >>>>>>> bug,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> >>>>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> >>>>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore
> .
> >> com
> >>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
> >> 1.9.0
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590
> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
> >>>>> release:
> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2631
> >> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is
> >> not
> >>>>>>>>>>> work
> >>>>>>>>>>>> (as
> >>>>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> >> appreciated.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> >>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix
> in?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2626
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2626
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked
> out
> >>>>> on
> >>>>>>>>>>> edit
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
> >>>>>>> forward
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com
> >
> >>>>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1519
> >> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1621
> >> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/commit/
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/commit/
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
> >>>>> thing
> >>>>>>> to
> >>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> criccomini@apache.org>
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
> >>>>> then.
> >>>>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can
> >> get
> >>>>>>> into
> >>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
> Bruin <
> >>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> >>>>>>> particular
> >>>>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
> tarball
> >>>>>>> than
> >>>>>>>>>>> to
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>>>>>>>>>> different?
> >>>>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
> >> Bruin <
> >>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
> >>>>> right
> >>>>>>>>>>> away?
> >>>>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
> planning
> >>>>> to
> >>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
> >> Once
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
> >>>>>>> picked
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to
> >> get
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
> >>>>> Riccomini
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are
> the
> >>>>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> Vulnerability in
> >>>>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> logging in
> >>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
> >>>>> local
> >>>>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> logger to
> >>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> LICENSE &
> >>>>>>> NOTICE
> >>>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> duplicate
> >>>>> and
> >>>>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
> >>>>> operate
> >>>>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> >>>>> shouldn't
> >>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
> >>>>> processes
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> displayed
> >>>>> over
> >>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
> running
> >>>>>>> task
> >>>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
> and
> >>>>> build
> >>>>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should
> not
> >>>>>>> push
> >>>>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size
> for
> >>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> >> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> unexpectedly
> >>>>>>>>>>> added to
> >>>>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>>>> cherry-picked
> >>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want
> >> in,
> >>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our
> >> dev
> >>>>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
> >> stability.
> >>>>>>> If
> >>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0
> >> test
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But 1765 is not a vulnerability, although *maybe* unexpected behaviour.

Bolke.


> On 30 Oct 2017, at 19:51, Chris Riccomini <cr...@apache.org> wrote:
> 
> I marked them as blocker. Still would like to understand how important this
> is, though. The API is experimental, so I could see an argument for them
> not being blocker provided the vulnerability is limited to experimental
> areas.
> 
> On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <as...@firemirror.com>
> wrote:
> 
>> I think we need to add
>> 
>> https://issues.apache.org/jira/browse/AIRFLOW-1764
>> https://issues.apache.org/jira/browse/AIRFLOW-1765
>> 
>> to that list as critical security fixes. I'll tackled 1765 tomorrow (UK
>> time) if no one gets round to it before then.
>> 
>> -ash
>> 
>>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org> wrote:
>>> 
>>> Hey all,
>>> 
>>> Current blockers for 1.9.0:
>>> 
>>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <criccomini@apache.org
>>> 
>>> wrote:
>>> 
>>>> Hey all,
>>>> 
>>>> Talked with @Bolke on Gitter. Here are the currently blocking issues for
>>>> 1.9.0:
>>>> 
>>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can
>>>> be False
>>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
>>>> config on PYTHONPATH
>>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck
>>>> in queued state
>>>> 
>>>> PRs are out for them. After these three are merged, I'm planning to cut
>> an
>>>> RC.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>>> 
>>>>> I upgraded our production environment today. Some observations
>>>>> 
>>>>> 1. Database migration (add max tries) didn’t workout of the box
>> correctly
>>>>> for Postgres. “max_tries = False” didn’t work and needed to be casted
>> to an
>>>>> int. Not sure how this gets to be false. This is a block imho
>>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
>> needed
>>>>> to rework quite a lot of dags
>>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>>>>> ‘data_profiler’ groups and thus should have automatic superuser
>> privileges.
>>>>> We don’t at the moment (eg. we cannot manage connections or see the
>> admin
>>>>> screens).
>>>>> 
>>>>> After fixing #1, the rest seems quite normal. Happy with the new
>> logging.
>>>>> 
>>>>> Bolke
>>>>> 
>>>>> 
>>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Bolke,
>>>>>> 
>>>>>> This one is a blocker: https://issues.apache.org/jira
>>>>> /browse/AIRFLOW-1731.
>>>>>> 
>>>>>> This requirers that we are sure that the /plugin/ folder is on the
>> path:
>>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>>> 
>>>>>> And maybe add an additional test and updating the docs a bit.
>> Hopefully
>>>>>> somewhere in the next few days, however I'm a bit busy with the Spark
>>>>>> Summit.
>>>>>> 
>>>>>> Cheers, Fokko
>>>>>> 
>>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>>> 
>>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>>> 
>>>>>>> What are the issues that are open?
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>> 
>>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> K, I will plan on cutting an alpha1 later this week. Something for
>> you
>>>>>>> guys to play with.
>>>>>>>> 
>>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>> hi Chris,
>>>>>>>> 
>>>>>>>> We are still running pre alpha. I am a bit preoccupied with
>> preparing
>>>>>>> for a conference and the team is readying a release of one of our
>> core
>>>>>>> products. So it will probably will be after this week when I get my
>>>>> hands
>>>>>>> dirty again.
>>>>>>>> 
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>> 
>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>> 
>>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
>> criccomini@apache.org
>>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> Checking in. Has anyone tested this in their environments? Stable
>>>>>>> releases
>>>>>>>>> depend on community involvement.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
>> alpha0,
>>>>>>> but
>>>>>>>>>> will be included in alpha1.
>>>>>>>>>> 
>>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
>> <fenglu@google.com.invalid
>>>>>> 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>> 
>>>>>>>>>>> I know it's annoying to have last minute commit com in, but this
>>>>> is a
>>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
>>>>>>> possible to
>>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>>> in?
>>>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>> 
>>>>>>>>>>> Feng
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hey all,
>>>>>>>>>>>> 
>>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>>>>>>> here:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>>> 
>>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>>> 
>>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>>>>>>>>>> 
>>>>>>>>>>>> The goal is to have the community install and run this to expose
>>>>> any
>>>>>>>>>>> bugs
>>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>>> 
>>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>>> 
>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>> are
>>>>>>>>>>> marked
>>>>>>>>>>>> as
>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>> exception
>>>>>>>>>>>> for
>>>>>>>>>>>> @on
>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>> to
>>>>>>>>>>> stdout
>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>> exception
>>>>>>>>>>> for
>>>>>>>>>>>> @once
>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>> to
>>>>>>> fail
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>>>> are
>>>>>>>>>>>> marked
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>>>> to
>>>>>>>>>>> fail
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut
>>>>> at
>>>>>>> RC
>>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>>>>>>> point of
>>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In addition the list below are mostly longer standing issues
>>>>> that
>>>>>>>>>>> are
>>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
>> 1525,
>>>>>>> 1258,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>>>>>>>>>> release,
>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>>>>>>> delay.
>>>>>>>>>>>> Here
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
>> SubDagOperator
>>>>>>> are
>>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
>> log
>>>>>>> to
>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>>> exception
>>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated
>> if
>>>>>>>>>>> Email
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
>> it
>>>>>>> to
>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
>>>>> the
>>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If you can help clean this up, that would be really
>>>>> appreciated.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
>>>>> SLA
>>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
>> major
>>>>>>> bug,
>>>>>>>>>>>> but
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.
>> com
>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
>> 1.9.0
>>>>>>>>>>> branch
>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>>>>> release:
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631
>> <
>>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is
>> not
>>>>>>>>>>> work
>>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
>> appreciated.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/pull/2626
>>>>> <
>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/pull/2626
>>>>> <
>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out
>>>>> on
>>>>>>>>>>> edit
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>>>>>>> forward
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> <
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> <
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/commit/
>>>>> <
>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
>> incubator-airflow/commit/
>>>>> <
>>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
>>>>> thing
>>>>>>> to
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
>>>>> then.
>>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can
>> get
>>>>>>> into
>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>>>>>> particular
>>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>>>>>>> than
>>>>>>>>>>> to
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
>> Bruin <
>>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
>>>>> right
>>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning
>>>>> to
>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
>> Once
>>>>>>> the
>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>>>>>>> picked
>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to
>> get
>>>>>>> the
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>>>>> Riccomini
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
>>>>> local
>>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>>>>>>> NOTICE
>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate
>>>>> and
>>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
>>>>> operate
>>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>>>> shouldn't
>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>>>> processes
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed
>>>>> over
>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>>>>>>> task
>>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
>>>>> build
>>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>>>>>>> push
>>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>>> cherry-picked
>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want
>> in,
>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our
>> dev
>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
>> stability.
>>>>>>> If
>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0
>> test
>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

I marked them as blocker. Still would like to understand how important this
is, though. The API is experimental, so I could see an argument for them
not being blocker provided the vulnerability is limited to experimental
areas.

On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <as...@firemirror.com>
wrote:

> I think we need to add
>
> https://issues.apache.org/jira/browse/AIRFLOW-1764
> https://issues.apache.org/jira/browse/AIRFLOW-1765
>
> to that list as critical security fixes. I'll tackled 1765 tomorrow (UK
> time) if no one gets round to it before then.
>
> -ash
>
> > On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org> wrote:
> >
> > Hey all,
> >
> > Current blockers for 1.9.0:
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-1711
> > https://issues.apache.org/jira/browse/AIRFLOW-1018
> >
> > Cheers,
> > Chris
> >
> > On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> Hey all,
> >>
> >> Talked with @Bolke on Gitter. Here are the currently blocking issues for
> >> 1.9.0:
> >>
> >> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can
> >> be False
> >> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
> >> config on PYTHONPATH
> >> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck
> >> in queued state
> >>
> >> PRs are out for them. After these three are merged, I'm planning to cut
> an
> >> RC.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >>> I upgraded our production environment today. Some observations
> >>>
> >>> 1. Database migration (add max tries) didn’t workout of the box
> correctly
> >>> for Postgres. “max_tries = False” didn’t work and needed to be casted
> to an
> >>> int. Not sure how this gets to be false. This is a block imho
> >>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
> needed
> >>> to rework quite a lot of dags
> >>> 3. We use LdAP for logins. We havent configured the ‘superuser’
> >>> ‘data_profiler’ groups and thus should have automatic superuser
> privileges.
> >>> We don’t at the moment (eg. we cannot manage connections or see the
> admin
> >>> screens).
> >>>
> >>> After fixing #1, the rest seems quite normal. Happy with the new
> logging.
> >>>
> >>> Bolke
> >>>
> >>>
> >>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
> >>> wrote:
> >>>>
> >>>> Hi Bolke,
> >>>>
> >>>> This one is a blocker: https://issues.apache.org/jira
> >>> /browse/AIRFLOW-1731.
> >>>>
> >>>> This requirers that we are sure that the /plugin/ folder is on the
> path:
> >>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>
> >>>> And maybe add an additional test and updating the docs a bit.
> Hopefully
> >>>> somewhere in the next few days, however I'm a bit busy with the Spark
> >>>> Summit.
> >>>>
> >>>> Cheers, Fokko
> >>>>
> >>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> >>>>
> >>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>>>>
> >>>>> What are the issues that are open?
> >>>>>
> >>>>> Cheers
> >>>>> Bolke
> >>>>>
> >>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>> K, I will plan on cutting an alpha1 later this week. Something for
> you
> >>>>> guys to play with.
> >>>>>>
> >>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
> >>>>> <ma...@gmail.com>> wrote:
> >>>>>> hi Chris,
> >>>>>>
> >>>>>> We are still running pre alpha. I am a bit preoccupied with
> preparing
> >>>>> for a conference and the team is readying a release of one of our
> core
> >>>>> products. So it will probably will be after this week when I get my
> >>> hands
> >>>>> dirty again.
> >>>>>>
> >>>>>> Cheers
> >>>>>> Bolke
> >>>>>>
> >>>>>> Verstuurd vanaf mijn iPad
> >>>>>>
> >>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> criccomini@apache.org
> >>>>> <ma...@apache.org>> het volgende geschreven:
> >>>>>>>
> >>>>>>> Hey all,
> >>>>>>>
> >>>>>>> Checking in. Has anyone tested this in their environments? Stable
> >>>>> releases
> >>>>>>> depend on community involvement.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Chris
> >>>>>>>
> >>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
> alpha0,
> >>>>> but
> >>>>>>>> will be included in alpha1.
> >>>>>>>>
> >>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> <fenglu@google.com.invalid
> >>>>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Chris,
> >>>>>>>>>
> >>>>>>>>> I know it's annoying to have last minute commit com in, but this
> >>> is a
> >>>>>>>>> highly desirable feature for folks using GCP operators, is it
> >>>>> possible to
> >>>>>>>>> include AIRFLOW-1635
> >>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
> >>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>> in?
> >>>>>>>>> More than happy to offer help if there's something I can do.
> >>>>>>>>> Thanks a lot.
> >>>>>>>>>
> >>>>>>>>> Feng
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hey all,
> >>>>>>>>>>
> >>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
> >>>>> here:
> >>>>>>>>>>
> >>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> >>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>
> >>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>
> >>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>>>>>>>
> >>>>>>>>>> The goal is to have the community install and run this to expose
> >>> any
> >>>>>>>>> bugs
> >>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>
> >>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>
> >>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> are
> >>>>>>>>> marked
> >>>>>>>>>> as
> >>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>>> exception
> >>>>>>>>>> for
> >>>>>>>>>> @on
> >>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
> to
> >>>>>>>>> stdout
> >>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>> exception
> >>>>>>>>> for
> >>>>>>>>>> @once
> >>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> to
> >>>>> fail
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Chris
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>
> >>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> >>> are
> >>>>>>>>>> marked
> >>>>>>>>>>>> as
> >>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> >>> to
> >>>>>>>>> fail
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut
> >>> at
> >>>>> RC
> >>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
> >>>>> point of
> >>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In addition the list below are mostly longer standing issues
> >>> that
> >>>>>>>>> are
> >>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
> 1525,
> >>>>> 1258,
> >>>>>>>>>> and
> >>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>>>>>>>> release,
> >>>>>>>>>> but
> >>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
> >>>>> delay.
> >>>>>>>>>> Here
> >>>>>>>>>>>>> are
> >>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> SubDagOperator
> >>>>> are
> >>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>>>>>>>> exception
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
> log
> >>>>> to
> >>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>>>>> exception
> >>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated
> if
> >>>>>>>>> Email
> >>>>>>>>>> is
> >>>>>>>>>>>>> Not
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
> it
> >>>>> to
> >>>>>>>>>> fail
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
> >>> the
> >>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you can help clean this up, that would be really
> >>> appreciated.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
> >>> SLA
> >>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
> major
> >>>>> bug,
> >>>>>>>>>> but
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
> >>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> >>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> >>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.
> com
> >>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
> 1.9.0
> >>>>>>>>> branch
> >>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
> >>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
> >>> release:
> >>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631
> <
> >>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is
> not
> >>>>>>>>> work
> >>>>>>>>>> (as
> >>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> appreciated.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> >>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2626
> >>> <
> >>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2626
> >>> <
> >>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out
> >>> on
> >>>>>>>>> edit
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
> >>>>> forward
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
> >>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> <
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> <
> >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/commit/
> >>> <
> >>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/commit/
> >>> <
> >>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
> >>> thing
> >>>>> to
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
> >>> then.
> >>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can
> get
> >>>>> into
> >>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> >>>>> particular
> >>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
> >>>>> than
> >>>>>>>>> to
> >>>>>>>>>> a
> >>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>>>>>>>> different?
> >>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
> Bruin <
> >>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
> >>> right
> >>>>>>>>> away?
> >>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning
> >>> to
> >>>>>>>>> cut
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
> Once
> >>>>> the
> >>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
> >>>>> picked
> >>>>>>>>>> into
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to
> get
> >>>>> the
> >>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
> >>> Riccomini
> >>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> >>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> >>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
> >>> local
> >>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
> >>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
> >>>>> NOTICE
> >>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate
> >>> and
> >>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
> >>> operate
> >>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> >>> shouldn't
> >>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
> >>> processes
> >>>>>>>>> can
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed
> >>> over
> >>>>>>>>> task
> >>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
> >>>>> task
> >>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
> >>> build
> >>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
> >>>>> push
> >>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
> >>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> >>>>>>>>> added to
> >>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>> cherry-picked
> >>>>>>>>> into
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want
> in,
> >>>>>>>>> please
> >>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our
> dev
> >>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
> stability.
> >>>>> If
> >>>>>>>>> you
> >>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0
> test
> >>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

I consider AIRFLOW-1765 to be invalid. Airflow ships out-of-the-box without authentication. Whether we keep it this way is up for discussion, but it is not a blocker.

Bolke


> On 30 Oct 2017, at 19:06, Ash Berlin-Taylor <as...@firemirror.com> wrote:
> 
> I think we need to add 
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1764
> https://issues.apache.org/jira/browse/AIRFLOW-1765 
> 
> to that list as critical security fixes. I'll tackled 1765 tomorrow (UK time) if no one gets round to it before then.
> 
> -ash
> 
>> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org> wrote:
>> 
>> Hey all,
>> 
>> Current blockers for 1.9.0:
>> 
>> https://issues.apache.org/jira/browse/AIRFLOW-1711
>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>> 
>> Cheers,
>> Chris
>> 
>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <cr...@apache.org>
>> wrote:
>> 
>>> Hey all,
>>> 
>>> Talked with @Bolke on Gitter. Here are the currently blocking issues for
>>> 1.9.0:
>>> 
>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can
>>> be False
>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
>>> config on PYTHONPATH
>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck
>>> in queued state
>>> 
>>> PRs are out for them. After these three are merged, I'm planning to cut an
>>> RC.
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>>> 
>>>> I upgraded our production environment today. Some observations
>>>> 
>>>> 1. Database migration (add max tries) didn’t workout of the box correctly
>>>> for Postgres. “max_tries = False” didn’t work and needed to be casted to an
>>>> int. Not sure how this gets to be false. This is a block imho
>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed
>>>> to rework quite a lot of dags
>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>>>> ‘data_profiler’ groups and thus should have automatic superuser privileges.
>>>> We don’t at the moment (eg. we cannot manage connections or see the admin
>>>> screens).
>>>> 
>>>> After fixing #1, the rest seems quite normal. Happy with the new logging.
>>>> 
>>>> Bolke
>>>> 
>>>> 
>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
>>>> wrote:
>>>>> 
>>>>> Hi Bolke,
>>>>> 
>>>>> This one is a blocker: https://issues.apache.org/jira
>>>> /browse/AIRFLOW-1731.
>>>>> 
>>>>> This requirers that we are sure that the /plugin/ folder is on the path:
>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>>> 
>>>>> And maybe add an additional test and updating the docs a bit. Hopefully
>>>>> somewhere in the next few days, however I'm a bit busy with the Spark
>>>>> Summit.
>>>>> 
>>>>> Cheers, Fokko
>>>>> 
>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>>> 
>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>>> 
>>>>>> What are the issues that are open?
>>>>>> 
>>>>>> Cheers
>>>>>> Bolke
>>>>>> 
>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>> K, I will plan on cutting an alpha1 later this week. Something for you
>>>>>> guys to play with.
>>>>>>> 
>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>>>>>> <ma...@gmail.com>> wrote:
>>>>>>> hi Chris,
>>>>>>> 
>>>>>>> We are still running pre alpha. I am a bit preoccupied with preparing
>>>>>> for a conference and the team is readying a release of one of our core
>>>>>> products. So it will probably will be after this week when I get my
>>>> hands
>>>>>> dirty again.
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>> 
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>> 
>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> Checking in. Has anyone tested this in their environments? Stable
>>>>>> releases
>>>>>>>> depend on community involvement.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>>>>>> but
>>>>>>>>> will be included in alpha1.
>>>>>>>>> 
>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fenglu@google.com.invalid
>>>>> 
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Chris,
>>>>>>>>>> 
>>>>>>>>>> I know it's annoying to have last minute commit com in, but this
>>>> is a
>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
>>>>>> possible to
>>>>>>>>>> include AIRFLOW-1635
>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>>> in?
>>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>>> Thanks a lot.
>>>>>>>>>> 
>>>>>>>>>> Feng
>>>>>>>>>> 
>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey all,
>>>>>>>>>>> 
>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>>>>>> here:
>>>>>>>>>>> 
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>>> 
>>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>>> 
>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>>>>>>>>> 
>>>>>>>>>>> The goal is to have the community install and run this to expose
>>>> any
>>>>>>>>>> bugs
>>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>>> 
>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>>> 
>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>>>>>>>> marked
>>>>>>>>>>> as
>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>> exception
>>>>>>>>>>> for
>>>>>>>>>>> @on
>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>>>>>>>>> stdout
>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>> exception
>>>>>>>>>> for
>>>>>>>>>>> @once
>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>>>>> fail
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>>> are
>>>>>>>>>>> marked
>>>>>>>>>>>>> as
>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>>> to
>>>>>>>>>> fail
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut
>>>> at
>>>>>> RC
>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>>>>>> point of
>>>>>>>>>>>>>> reference.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In addition the list below are mostly longer standing issues
>>>> that
>>>>>>>>>> are
>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>>>>>> 1258,
>>>>>>>>>>> and
>>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>>>>>>>>> release,
>>>>>>>>>>> but
>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>>>>>> delay.
>>>>>>>>>>> Here
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>>>>> are
>>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>>>>>> exception
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>>>>>> to
>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>>> exception
>>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>>>>>>>>> Email
>>>>>>>>>>> is
>>>>>>>>>>>>>> Not
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>>>>> to
>>>>>>>>>>> fail
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
>>>> the
>>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you can help clean this up, that would be really
>>>> appreciated.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
>>>> SLA
>>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a major
>>>>>> bug,
>>>>>>>>>>> but
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com
>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>>>>>>>>> branch
>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>>>> release:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>>>>>>>>> work
>>>>>>>>>>> (as
>>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>>>> <
>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>>>> <
>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out
>>>> on
>>>>>>>>>> edit
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>>>>>> forward
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>> <
>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>> <
>>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
>>>> thing
>>>>>> to
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
>>>> then.
>>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>>>>>> into
>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>>>>> particular
>>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>>>>>> than
>>>>>>>>>> to
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>>>>>>>>> different?
>>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
>>>> right
>>>>>>>>>> away?
>>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning
>>>> to
>>>>>>>>>> cut
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>>>>>> the
>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>>>>>> picked
>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>>>>>> the
>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>>>> Riccomini
>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
>>>> local
>>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>>>>>> NOTICE
>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate
>>>> and
>>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
>>>> operate
>>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>>> shouldn't
>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>>> processes
>>>>>>>>>> can
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed
>>>> over
>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>>>>>> task
>>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
>>>> build
>>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>>>>>> push
>>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>>>>>>>>> added to
>>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>>> cherry-picked
>>>>>>>>>> into
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>>>>>>>>> please
>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>>>>>> If
>>>>>>>>>> you
>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>

Re: Airflow 1.9.0 status

Posted by Ash Berlin-Taylor <as...@firemirror.com>.

I think we need to add 

https://issues.apache.org/jira/browse/AIRFLOW-1764
https://issues.apache.org/jira/browse/AIRFLOW-1765 

to that list as critical security fixes. I'll tackled 1765 tomorrow (UK time) if no one gets round to it before then.

-ash

> On 30 Oct 2017, at 17:58, Chris Riccomini <cr...@apache.org> wrote:
> 
> Hey all,
> 
> Current blockers for 1.9.0:
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1711
> https://issues.apache.org/jira/browse/AIRFLOW-1018
> 
> Cheers,
> Chris
> 
> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <cr...@apache.org>
> wrote:
> 
>> Hey all,
>> 
>> Talked with @Bolke on Gitter. Here are the currently blocking issues for
>> 1.9.0:
>> 
>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can
>> be False
>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
>> config on PYTHONPATH
>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck
>> in queued state
>> 
>> PRs are out for them. After these three are merged, I'm planning to cut an
>> RC.
>> 
>> Cheers,
>> Chris
>> 
>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>> 
>>> I upgraded our production environment today. Some observations
>>> 
>>> 1. Database migration (add max tries) didn’t workout of the box correctly
>>> for Postgres. “max_tries = False” didn’t work and needed to be casted to an
>>> int. Not sure how this gets to be false. This is a block imho
>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed
>>> to rework quite a lot of dags
>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>>> ‘data_profiler’ groups and thus should have automatic superuser privileges.
>>> We don’t at the moment (eg. we cannot manage connections or see the admin
>>> screens).
>>> 
>>> After fixing #1, the rest seems quite normal. Happy with the new logging.
>>> 
>>> Bolke
>>> 
>>> 
>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
>>> wrote:
>>>> 
>>>> Hi Bolke,
>>>> 
>>>> This one is a blocker: https://issues.apache.org/jira
>>> /browse/AIRFLOW-1731.
>>>> 
>>>> This requirers that we are sure that the /plugin/ folder is on the path:
>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>>> 
>>>> And maybe add an additional test and updating the docs a bit. Hopefully
>>>> somewhere in the next few days, however I'm a bit busy with the Spark
>>>> Summit.
>>>> 
>>>> Cheers, Fokko
>>>> 
>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>>>> 
>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>>>> 
>>>>> What are the issues that are open?
>>>>> 
>>>>> Cheers
>>>>> Bolke
>>>>> 
>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>>>>> 
>>>>>> K, I will plan on cutting an alpha1 later this week. Something for you
>>>>> guys to play with.
>>>>>> 
>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>> hi Chris,
>>>>>> 
>>>>>> We are still running pre alpha. I am a bit preoccupied with preparing
>>>>> for a conference and the team is readying a release of one of our core
>>>>> products. So it will probably will be after this week when I get my
>>> hands
>>>>> dirty again.
>>>>>> 
>>>>>> Cheers
>>>>>> Bolke
>>>>>> 
>>>>>> Verstuurd vanaf mijn iPad
>>>>>> 
>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>>>>> <ma...@apache.org>> het volgende geschreven:
>>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> Checking in. Has anyone tested this in their environments? Stable
>>>>> releases
>>>>>>> depend on community involvement.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>>>>> but
>>>>>>>> will be included in alpha1.
>>>>>>>> 
>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fenglu@google.com.invalid
>>>> 
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Chris,
>>>>>>>>> 
>>>>>>>>> I know it's annoying to have last minute commit com in, but this
>>> is a
>>>>>>>>> highly desirable feature for folks using GCP operators, is it
>>>>> possible to
>>>>>>>>> include AIRFLOW-1635
>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>>>>> in?
>>>>>>>>> More than happy to offer help if there's something I can do.
>>>>>>>>> Thanks a lot.
>>>>>>>>> 
>>>>>>>>> Feng
>>>>>>>>> 
>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey all,
>>>>>>>>>> 
>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>>>>> here:
>>>>>>>>>> 
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>>>> airflow/1.9.0alpha0/>
>>>>>>>>>> 
>>>>>>>>>> The bin tarball can be installed with pip:
>>>>>>>>>> 
>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>>>>>>>> 
>>>>>>>>>> The goal is to have the community install and run this to expose
>>> any
>>>>>>>>> bugs
>>>>>>>>>> before we move on to official release candidates.
>>>>>>>>>> 
>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>>>>> 
>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>>>>>>> marked
>>>>>>>>>> as
>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>> exception
>>>>>>>>>> for
>>>>>>>>>> @on
>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>>>>>>>> stdout
>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>> exception
>>>>>>>>> for
>>>>>>>>>> @once
>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>>>> fail
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>>>>> 
>>>>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>>>>> 
>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>> are
>>>>>>>>>> marked
>>>>>>>>>>>> as
>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>> to
>>>>>>>>> fail
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Chris
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut
>>> at
>>>>> RC
>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>>>>> point of
>>>>>>>>>>>>> reference.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In addition the list below are mostly longer standing issues
>>> that
>>>>>>>>> are
>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>>>>> 1258,
>>>>>>>>>> and
>>>>>>>>>>>>> 976 as blocker?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>>>>>>>> release,
>>>>>>>>>> but
>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>>>>> delay.
>>>>>>>>>> Here
>>>>>>>>>>>>> are
>>>>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>>>> are
>>>>>>>>>>>>> marked as
>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>>>>> exception
>>>>>>>>>>>>> for
>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>>>>> to
>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>>>>> exception
>>>>>>>>>>>>> for @once
>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>>>>>>>> Email
>>>>>>>>>> is
>>>>>>>>>>>>> Not
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>>>> to
>>>>>>>>>> fail
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
>>> the
>>>>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you can help clean this up, that would be really
>>> appreciated.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
>>> SLA
>>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a major
>>>>> bug,
>>>>>>>>>> but
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>>>>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>>>>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>>>> http://www.simpli.fi/>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com
>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>>>>>>>> branch
>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>>>>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>>> release:
>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>>>>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>>>>>>>> work
>>>>>>>>>> (as
>>>>>>>>>>>>>>>> far
>>>>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>>>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>>> <
>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>>> <
>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out
>>> on
>>>>>>>>> edit
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>>>>> forward
>>>>>>>>> to
>>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>> <
>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>> <
>>>>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
>>> thing
>>>>> to
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
>>> then.
>>>>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>>>>> into
>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>>>> particular
>>>>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>>>>> than
>>>>>>>>> to
>>>>>>>>>> a
>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>>>>>>>> different?
>>>>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
>>> right
>>>>>>>>> away?
>>>>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning
>>> to
>>>>>>>>> cut
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>>>>> the
>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>>>>> picked
>>>>>>>>>> into
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>>>>> the
>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>>> Riccomini
>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>>>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>>>>>>>> Variable
>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
>>> local
>>>>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>>>>> NOTICE
>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate
>>> and
>>>>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
>>> operate
>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>>> shouldn't
>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>>> processes
>>>>>>>>> can
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed
>>> over
>>>>>>>>> task
>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>>>>> task
>>>>>>>>>>>>>>>> causes
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
>>> build
>>>>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>>>>> push
>>>>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>>>>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>>>>>>>> added to
>>>>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>>> cherry-picked
>>>>>>>>> into
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>>>>>>>> please
>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>>>>> If
>>>>>>>>> you
>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

Current blockers for 1.9.0:

https://issues.apache.org/jira/browse/AIRFLOW-1711
https://issues.apache.org/jira/browse/AIRFLOW-1018

Cheers,
Chris

On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Hey all,
>
> Talked with @Bolke on Gitter. Here are the currently blocking issues for
> 1.9.0:
>
> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can
> be False
> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
> config on PYTHONPATH
> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck
> in queued state
>
> PRs are out for them. After these three are merged, I'm planning to cut an
> RC.
>
> Cheers,
> Chris
>
> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> I upgraded our production environment today. Some observations
>>
>> 1. Database migration (add max tries) didn’t workout of the box correctly
>> for Postgres. “max_tries = False” didn’t work and needed to be casted to an
>> int. Not sure how this gets to be false. This is a block imho
>> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed
>> to rework quite a lot of dags
>> 3. We use LdAP for logins. We havent configured the ‘superuser’
>> ‘data_profiler’ groups and thus should have automatic superuser privileges.
>> We don’t at the moment (eg. we cannot manage connections or see the admin
>> screens).
>>
>> After fixing #1, the rest seems quite normal. Happy with the new logging.
>>
>> Bolke
>>
>>
>> > On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>> >
>> > Hi Bolke,
>> >
>> > This one is a blocker: https://issues.apache.org/jira
>> /browse/AIRFLOW-1731.
>> >
>> > This requirers that we are sure that the /plugin/ folder is on the path:
>> > https://github.com/apache/incubator-airflow/blob/eb2f589099b
>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>> >
>> > And maybe add an additional test and updating the docs a bit. Hopefully
>> > somewhere in the next few days, however I'm a bit busy with the Spark
>> > Summit.
>> >
>> > Cheers, Fokko
>> >
>> > 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>> >
>> >> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>> >>
>> >> What are the issues that are open?
>> >>
>> >> Cheers
>> >> Bolke
>> >>
>> >>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
>> wrote:
>> >>>
>> >>> K, I will plan on cutting an alpha1 later this week. Something for you
>> >> guys to play with.
>> >>>
>> >>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>> >> <ma...@gmail.com>> wrote:
>> >>> hi Chris,
>> >>>
>> >>> We are still running pre alpha. I am a bit preoccupied with preparing
>> >> for a conference and the team is readying a release of one of our core
>> >> products. So it will probably will be after this week when I get my
>> hands
>> >> dirty again.
>> >>>
>> >>> Cheers
>> >>> Bolke
>> >>>
>> >>> Verstuurd vanaf mijn iPad
>> >>>
>> >>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>> >> <ma...@apache.org>> het volgende geschreven:
>> >>>>
>> >>>> Hey all,
>> >>>>
>> >>>> Checking in. Has anyone tested this in their environments? Stable
>> >> releases
>> >>>> depend on community involvement.
>> >>>>
>> >>>> Cheers,
>> >>>> Chris
>> >>>>
>> >>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>> >> criccomini@apache.org <ma...@apache.org>>
>> >>>> wrote:
>> >>>>
>> >>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>> >> but
>> >>>>> will be included in alpha1.
>> >>>>>
>> >>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fenglu@google.com.invalid
>> >
>> >> wrote:
>> >>>>>>
>> >>>>>> Hi Chris,
>> >>>>>>
>> >>>>>> I know it's annoying to have last minute commit com in, but this
>> is a
>> >>>>>> highly desirable feature for folks using GCP operators, is it
>> >> possible to
>> >>>>>> include AIRFLOW-1635
>> >>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>> >> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>> >>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>> >>>>>> in?
>> >>>>>> More than happy to offer help if there's something I can do.
>> >>>>>> Thanks a lot.
>> >>>>>>
>> >>>>>> Feng
>> >>>>>>
>> >>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>> >> criccomini@apache.org <ma...@apache.org>>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Hey all,
>> >>>>>>>
>> >>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>> >> here:
>> >>>>>>>
>> >>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> >> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>> >> airflow/1.9.0alpha0/>
>> >>>>>>>
>> >>>>>>> The bin tarball can be installed with pip:
>> >>>>>>>
>> >>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>> >>>>>>>
>> >>>>>>> The goal is to have the community install and run this to expose
>> any
>> >>>>>> bugs
>> >>>>>>> before we move on to official release candidates.
>> >>>>>>>
>> >>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>> >>>>>>>
>> >>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> >>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> >>>>>> marked
>> >>>>>>> as
>> >>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>> >> exception
>> >>>>>>> for
>> >>>>>>> @on
>> >>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>> >>>>>> stdout
>> >>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>> exception
>> >>>>>> for
>> >>>>>>> @once
>> >>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>> >> fail
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>> Chris
>> >>>>>>>
>> >>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>> >> criccomini@apache.org <ma...@apache.org>
>> >>>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>> >>>>>>>>
>> >>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>> >>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Works for me. Will try and cut a beta before end of week.
>> >>>>>>>>>
>> >>>>>>>>> Blockers for 1.9.0 are:
>> >>>>>>>>>
>> >>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> >>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> >>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>> are
>> >>>>>>> marked
>> >>>>>>>>> as
>> >>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>> to
>> >>>>>> fail
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>> >> bdbruin@gmail.com <ma...@gmail.com>>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi Chris
>> >>>>>>>>>>
>> >>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut
>> at
>> >> RC
>> >>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>> >> point of
>> >>>>>>>>>> reference.
>> >>>>>>>>>>
>> >>>>>>>>>> In addition the list below are mostly longer standing issues
>> that
>> >>>>>> are
>> >>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>> >> 1258,
>> >>>>>>> and
>> >>>>>>>>>> 976 as blocker?
>> >>>>>>>>>>
>> >>>>>>>>>> Cheers
>> >>>>>>>>>> Bolke
>> >>>>>>>>>>
>> >>>>>>>>>> Verstuurd vanaf mijn iPad
>> >>>>>>>>>>
>> >>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>> >>>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>>>> het volgende geschreven:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hey all,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>> >>>>>> release,
>> >>>>>>> but
>> >>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>> >> delay.
>> >>>>>>> Here
>> >>>>>>>>>> are
>> >>>>>>>>>>> the bugs that I'm tracking:
>> >>>>>>>>>>>
>> >>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> >>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> >>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>> >> are
>> >>>>>>>>>> marked as
>> >>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>> >>>>>>> exception
>> >>>>>>>>>> for
>> >>>>>>>>>>> @on
>> >>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>> >> to
>> >>>>>>>>>> stdout
>> >>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>> >>>>>> exception
>> >>>>>>>>>> for @once
>> >>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>> >>>>>> Email
>> >>>>>>> is
>> >>>>>>>>>> Not
>> >>>>>>>>>>> be
>> >>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>> >> to
>> >>>>>>> fail
>> >>>>>>>>>>>
>> >>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
>> the
>> >>>>>>>>>>> v1-9-stable and beta release.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If you can help clean this up, that would be really
>> appreciated.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Cheers,
>> >>>>>>>>>>> Chris
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Marked it for 1.9.0.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>> >>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
>> SLA
>> >>>>>>>>>> callbacks
>> >>>>>>>>>>>>> are not working correctly without emails... Its not a major
>> >> bug,
>> >>>>>>> but
>> >>>>>>>>>> it
>> >>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Link to Jira:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>> >> https://issues.apache.org/jira/browse/AIRFLOW-988>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Link to PR:
>> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>> >> https://github.com/apache/incubator-airflow/pull/2415>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks!
>> >>>>>>>>>>>>> Charlie Jones
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> CHARLIE JONES
>> >>>>>>>>>>>>> Data Engineer
>> >>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>> >> 972.821.7631 <tel:972.821.7631>
>> >>>>>>>>>>>>> __________________________________________________
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Programmatic Performance.* Localized.*
>> >>>>>>>>>>>>> __________________________________________________
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> >>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>> >> http://www.simpli.fi/>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Merged.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> >>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com
>> >>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>> >>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>> >>>>>> branch
>> >>>>>>>>>> due
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>> this issue.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>> >> https://github.com/apache/incubator-airflow/pull/2590>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> Ryan
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> >>>>>>>>>>>>> <fokko@driesprong.frl
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hi All,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
>> release:
>> >>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>> >> https://github.com/apache/incubator-airflow/pull/2631>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>> >>>>>> work
>> >>>>>>> (as
>> >>>>>>>>>>>>> far
>> >>>>>>>>>>>>>>> as I
>> >>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Cheers, Fokko
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>> >>>>>>> criccomini@apache.org <ma...@apache.org>
>> >>>>>>>>>>> :
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Done!
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> >>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>> >> michael.crawford@modernizingmedicine.com>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>> <
>> >> https://github.com/apache/incubator-airflow/pull/2626> <
>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
>> <
>> >> https://github.com/apache/incubator-airflow/pull/2626>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out
>> on
>> >>>>>> edit
>> >>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>>> types.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>>>> Mike
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>> >> forward
>> >>>>>> to
>> >>>>>>>>>>>>>> these
>> >>>>>>>>>>>>>>>>> fixes!
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>> >>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>> >>>>>>>>>>>>>>>> .
>> >>>>>>>>>>>>>>>>>> invalid
>> >>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Can we get this in?
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>> >> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>> >> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> <
>> >> https://github.com/apache/incubator-airflow/commit/>
>> >>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> <
>> >> https://github.com/apache/incubator-airflow/commit/>
>> >>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
>> thing
>> >> to
>> >>>>>>>>>>>>> have
>> >>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>> 1.9.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>> >>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
>> then.
>> >>>>>>>>>>>>> Initial
>> >>>>>>>>>>>>>>>>> warning
>> >>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>> >> into
>> >>>>>>>>>>>>> 1.9.0
>> >>>>>>>>>>>>>> at
>> >>>>>>>>>>>>>>>>> that
>> >>>>>>>>>>>>>>>>>>>>> point.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>> >>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>> >>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>> >> particular
>> >>>>>>>>>>>>> fixed
>> >>>>>>>>>>>>>>>> point
>> >>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>> >> than
>> >>>>>> to
>> >>>>>>> a
>> >>>>>>>>>>>>>> git
>> >>>>>>>>>>>>>>>>> pull.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Bolke
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>> >>>>>> different?
>> >>>>>>>>>>>>>> IIRC,
>> >>>>>>>>>>>>>>>> it's
>> >>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>> >>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
>> right
>> >>>>>> away?
>> >>>>>>>>>>>>>>> Isn’t a
>> >>>>>>>>>>>>>>>>>>>> beta
>> >>>>>>>>>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>> >>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning
>> to
>> >>>>>> cut
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>> stable
>> >>>>>>>>>>>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>> >> the
>> >>>>>>>>>>>>> stable
>> >>>>>>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>>>>>>>>>> cut, I
>> >>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>> >> picked
>> >>>>>>> into
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> branch,
>> >>>>>>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>> >> the
>> >>>>>>>>>>>>> release
>> >>>>>>>>>>>>>>>> out.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
>> Riccomini
>> >> <
>> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
>> criccomini@apache.org>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>> >>>>>>>>>>>>> outstanding
>> >>>>>>>>>>>>>> PRs
>> >>>>>>>>>>>>>>>>> that
>> >>>>>>>>>>>>>>>>>>>>> are
>> >>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>> >>>>>> Variable
>> >>>>>>>>>>>>>>>> endpoint
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>> >>>>>> Airflow
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
>> local
>> >>>>>>>>>>>>> loggers
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
>> log
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>> >> NOTICE
>> >>>>>>>>>>>>> issue
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate
>> and
>> >>>>>>>>>>>>> unneeded
>> >>>>>>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
>> operate
>> >>>>>> HDFS
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>> >>>>>> |airflow/jobs.py:create_dag_ru
>> >>>>>>>>>>>>> n()
>> >>>>>>>>>>>>>>>>>>>> exception
>> >>>>>>>>>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>>>>>>>>> @on
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
>> shouldn't
>> >>>>>>>>>>>>> include
>> >>>>>>>>>>>>>>>> paused
>> >>>>>>>>>>>>>>>>>>>>> DAGs
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
>> processes
>> >>>>>> can
>> >>>>>>>>>>>>> not
>> >>>>>>>>>>>>>> log
>> >>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>> stdout
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed
>> over
>> >>>>>> task
>> >>>>>>>>>>>>>>>> instances
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>> >>>>>> |airflow/jobs.py:manage_slas()
>> >>>>>>>>>>>>>>>> exception
>> >>>>>>>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>>>>>>>>> @once
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>> >> task
>> >>>>>>>>>>>>> causes
>> >>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>> fail
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>> >>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>> >>>>>>>>>>>>>>>>>>>>>>>> examples
>> >>>>>>>>>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>> >>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>> >>>>>>>>>>>>>>>>>>>> job
>> >>>>>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>>> real
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
>> build
>> >>>>>>>>>>>>> matrix
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>> >> push
>> >>>>>>>>>>>>> XComs
>> >>>>>>>>>>>>>> by
>> >>>>>>>>>>>>>>>>>>>> default
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>> >> XComs
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>> >>>>>> added to
>> >>>>>>>>>>>>>> hive
>> >>>>>>>>>>>>>>>> conf
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
>> cherry-picked
>> >>>>>> into
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>> 1.9.0
>> >>>>>>>>>>>>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>> >>>>>> please
>> >>>>>>>>>>>>> set
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>> fix
>> >>>>>>>>>>>>>>>>>>>>>>>> version
>> >>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>> >>>>>>> cluster,
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>> has
>> >>>>>>>>>>>>>>>>>>>>>> been
>> >>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>> >> If
>> >>>>>> you
>> >>>>>>>>>>>>> run
>> >>>>>>>>>>>>>>>>>>>> Airflow,
>> >>>>>>>>>>>>>>>>>>>>>> it's
>> >>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>> >>>>>> branch
>> >>>>>>>>>>>>>>>> somewhere,
>> >>>>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>> verify
>> >>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>
>> >>
>>
>>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

Talked with @Bolke on Gitter. Here are the currently blocking issues for
1.9.0:

AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can be
False
AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
config on PYTHONPATH
AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck in
queued state

PRs are out for them. After these three are merged, I'm planning to cut an
RC.

Cheers,
Chris

On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> I upgraded our production environment today. Some observations
>
> 1. Database migration (add max tries) didn’t workout of the box correctly
> for Postgres. “max_tries = False” didn’t work and needed to be casted to an
> int. Not sure how this gets to be false. This is a block imho
> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed
> to rework quite a lot of dags
> 3. We use LdAP for logins. We havent configured the ‘superuser’
> ‘data_profiler’ groups and thus should have automatic superuser privileges.
> We don’t at the moment (eg. we cannot manage connections or see the admin
> screens).
>
> After fixing #1, the rest seems quite normal. Happy with the new logging.
>
> Bolke
>
>
> > On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> >
> > Hi Bolke,
> >
> > This one is a blocker: https://issues.apache.org/
> jira/browse/AIRFLOW-1731.
> >
> > This requirers that we are sure that the /plugin/ folder is on the path:
> > https://github.com/apache/incubator-airflow/blob/
> eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >
> > And maybe add an additional test and updating the docs a bit. Hopefully
> > somewhere in the next few days, however I'm a bit busy with the Spark
> > Summit.
> >
> > Cheers, Fokko
> >
> > 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> >
> >> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>
> >> What are the issues that are open?
> >>
> >> Cheers
> >> Bolke
> >>
> >>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org>
> wrote:
> >>>
> >>> K, I will plan on cutting an alpha1 later this week. Something for you
> >> guys to play with.
> >>>
> >>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>> hi Chris,
> >>>
> >>> We are still running pre alpha. I am a bit preoccupied with preparing
> >> for a conference and the team is readying a release of one of our core
> >> products. So it will probably will be after this week when I get my
> hands
> >> dirty again.
> >>>
> >>> Cheers
> >>> Bolke
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
> >>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
> >> <ma...@apache.org>> het volgende geschreven:
> >>>>
> >>>> Hey all,
> >>>>
> >>>> Checking in. Has anyone tested this in their environments? Stable
> >> releases
> >>>> depend on community involvement.
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >> criccomini@apache.org <ma...@apache.org>>
> >>>> wrote:
> >>>>
> >>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
> >> but
> >>>>> will be included in alpha1.
> >>>>>
> >>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid>
> >> wrote:
> >>>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> I know it's annoying to have last minute commit com in, but this is
> a
> >>>>>> highly desirable feature for folks using GCP operators, is it
> >> possible to
> >>>>>> include AIRFLOW-1635
> >>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
> >> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>> in?
> >>>>>> More than happy to offer help if there's something I can do.
> >>>>>> Thanks a lot.
> >>>>>>
> >>>>>> Feng
> >>>>>>
> >>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >> criccomini@apache.org <ma...@apache.org>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey all,
> >>>>>>>
> >>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
> >> here:
> >>>>>>>
> >>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> >> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> >> airflow/1.9.0alpha0/>
> >>>>>>>
> >>>>>>> The bin tarball can be installed with pip:
> >>>>>>>
> >>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>>>>
> >>>>>>> The goal is to have the community install and run this to expose
> any
> >>>>>> bugs
> >>>>>>> before we move on to official release candidates.
> >>>>>>>
> >>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>
> >>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>>>>> marked
> >>>>>>> as
> >>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >> exception
> >>>>>>> for
> >>>>>>> @on
> >>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>>>>> stdout
> >>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> >>>>>> for
> >>>>>>> @once
> >>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >> fail
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Chris
> >>>>>>>
> >>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >> criccomini@apache.org <ma...@apache.org>
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>>>>
> >>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>>>>
> >>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>
> >>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> are
> >>>>>>> marked
> >>>>>>>>> as
> >>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >>>>>> fail
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Chris
> >>>>>>>>>>
> >>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut at
> >> RC
> >>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
> >> point of
> >>>>>>>>>> reference.
> >>>>>>>>>>
> >>>>>>>>>> In addition the list below are mostly longer standing issues
> that
> >>>>>> are
> >>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
> >> 1258,
> >>>>>>> and
> >>>>>>>>>> 976 as blocker?
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>> Bolke
> >>>>>>>>>>
> >>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>
> >>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>
> >>>>>>>>>>> Hey all,
> >>>>>>>>>>>
> >>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>>>>> release,
> >>>>>>> but
> >>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
> >> delay.
> >>>>>>> Here
> >>>>>>>>>> are
> >>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>
> >>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> >> are
> >>>>>>>>>> marked as
> >>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>>>>> exception
> >>>>>>>>>> for
> >>>>>>>>>>> @on
> >>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
> >> to
> >>>>>>>>>> stdout
> >>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>> exception
> >>>>>>>>>> for @once
> >>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
> >>>>>> Email
> >>>>>>> is
> >>>>>>>>>> Not
> >>>>>>>>>>> be
> >>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> >> to
> >>>>>>> fail
> >>>>>>>>>>>
> >>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut
> the
> >>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>
> >>>>>>>>>>> If you can help clean this up, that would be really
> appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Chris
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
> SLA
> >>>>>>>>>> callbacks
> >>>>>>>>>>>>> are not working correctly without emails... Its not a major
> >> bug,
> >>>>>>> but
> >>>>>>>>>> it
> >>>>>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> >> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
> >> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> >> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> >> http://www.simpli.fi/>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com
> >>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
> >>>>>> branch
> >>>>>>>>>> due
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
> >> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
> >> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
> >>>>>> work
> >>>>>>> (as
> >>>>>>>>>>>>> far
> >>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> >> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> >> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> >> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
> >>>>>> edit
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
> >> forward
> >>>>>> to
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
> >>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
> >> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
> >> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
> >> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
> >> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing
> >> to
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
> then.
> >>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
> >> into
> >>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> >> particular
> >>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
> >> than
> >>>>>> to
> >>>>>>> a
> >>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>>>>> different?
> >>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> >>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> >>>>>> away?
> >>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
> >>>>>> cut
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
> >> the
> >>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
> >> picked
> >>>>>>> into
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
> >> the
> >>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
> >> <
> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org
> >
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> >>>>>> Variable
> >>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> >>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> >>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to
> log
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
> >> NOTICE
> >>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> >>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> >>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> shouldn't
> >>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
> >>>>>> can
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> >>>>>> task
> >>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
> >> task
> >>>>>>>>>>>>> causes
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and
> build
> >>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
> >> push
> >>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
> >> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> >>>>>> added to
> >>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> cherry-picked
> >>>>>> into
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> >>>>>> please
> >>>>>>>>>>>>> set
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> >>>>>>> cluster,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
> >> If
> >>>>>> you
> >>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> >>>>>> branch
> >>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

I upgraded our production environment today. Some observations

1. Database migration (add max tries) didn’t workout of the box correctly for Postgres. “max_tries = False” didn’t work and needed to be casted to an int. Not sure how this gets to be false. This is a block imho
2. SSHExecuteOperator not having backwards compatibility sucks. We needed to rework quite a lot of dags
3. We use LdAP for logins. We havent configured the ‘superuser’ ‘data_profiler’ groups and thus should have automatic superuser privileges. We don’t at the moment (eg. we cannot manage connections or see the admin screens).

After fixing #1, the rest seems quite normal. Happy with the new logging.

Bolke


> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fo...@driesprong.frl> wrote:
> 
> Hi Bolke,
> 
> This one is a blocker: https://issues.apache.org/jira/browse/AIRFLOW-1731.
> 
> This requirers that we are sure that the /plugin/ folder is on the path:
> https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> 
> And maybe add an additional test and updating the docs a bit. Hopefully
> somewhere in the next few days, however I'm a bit busy with the Spark
> Summit.
> 
> Cheers, Fokko
> 
> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> 
>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>> 
>> What are the issues that are open?
>> 
>> Cheers
>> Bolke
>> 
>>> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org> wrote:
>>> 
>>> K, I will plan on cutting an alpha1 later this week. Something for you
>> guys to play with.
>>> 
>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>> <ma...@gmail.com>> wrote:
>>> hi Chris,
>>> 
>>> We are still running pre alpha. I am a bit preoccupied with preparing
>> for a conference and the team is readying a release of one of our core
>> products. So it will probably will be after this week when I get my hands
>> dirty again.
>>> 
>>> Cheers
>>> Bolke
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>> <ma...@apache.org>> het volgende geschreven:
>>>> 
>>>> Hey all,
>>>> 
>>>> Checking in. Has anyone tested this in their environments? Stable
>> releases
>>>> depend on community involvement.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>> criccomini@apache.org <ma...@apache.org>>
>>>> wrote:
>>>> 
>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>> but
>>>>> will be included in alpha1.
>>>>> 
>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid>
>> wrote:
>>>>>> 
>>>>>> Hi Chris,
>>>>>> 
>>>>>> I know it's annoying to have last minute commit com in, but this is a
>>>>>> highly desirable feature for folks using GCP operators, is it
>> possible to
>>>>>> include AIRFLOW-1635
>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>> in?
>>>>>> More than happy to offer help if there's something I can do.
>>>>>> Thanks a lot.
>>>>>> 
>>>>>> Feng
>>>>>> 
>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>> criccomini@apache.org <ma...@apache.org>>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>> here:
>>>>>>> 
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>> airflow/1.9.0alpha0/>
>>>>>>> 
>>>>>>> The bin tarball can be installed with pip:
>>>>>>> 
>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>>>>> 
>>>>>>> The goal is to have the community install and run this to expose any
>>>>>> bugs
>>>>>>> before we move on to official release candidates.
>>>>>>> 
>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>> 
>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>>>> marked
>>>>>>> as
>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>> exception
>>>>>>> for
>>>>>>> @on
>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>>>>> stdout
>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>>>>> for
>>>>>>> @once
>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>> fail
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>> criccomini@apache.org <ma...@apache.org>
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>>>>>>> 
>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>> 
>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>> 
>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>>>>> marked
>>>>>>>>> as
>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>>>>> fail
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Chris
>>>>>>>>>> 
>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut at
>> RC
>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>> point of
>>>>>>>>>> reference.
>>>>>>>>>> 
>>>>>>>>>> In addition the list below are mostly longer standing issues that
>>>>>> are
>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>> 1258,
>>>>>>> and
>>>>>>>>>> 976 as blocker?
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> Bolke
>>>>>>>>>> 
>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>> 
>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>> 
>>>>>>>>>>> Hey all,
>>>>>>>>>>> 
>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>>>>> release,
>>>>>>> but
>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>> delay.
>>>>>>> Here
>>>>>>>>>> are
>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>> 
>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>> are
>>>>>>>>>> marked as
>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>> exception
>>>>>>>>>> for
>>>>>>>>>>> @on
>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>> to
>>>>>>>>>> stdout
>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>> exception
>>>>>>>>>> for @once
>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>>>>> Email
>>>>>>> is
>>>>>>>>>> Not
>>>>>>>>>>> be
>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>> to
>>>>>>> fail
>>>>>>>>>>> 
>>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut the
>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>> 
>>>>>>>>>>> If you can help clean this up, that would be really appreciated.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>>>>> cjones@simpli.fi <ma...@simpli.fi>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>>>>>>>>>> callbacks
>>>>>>>>>>>>> are not working correctly without emails... Its not a major
>> bug,
>>>>>>> but
>>>>>>>>>> it
>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>> 
>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>> http://www.simpli.fi/>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>>>>> ryan.buckley@bluecore.com <ma...@bluecore.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>>>>> branch
>>>>>>>>>> due
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>>>>> work
>>>>>>> (as
>>>>>>>>>>>>> far
>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>> :
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
>>>>>> edit
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>> forward
>>>>>> to
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing
>> to
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>> into
>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>> particular
>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>> than
>>>>>> to
>>>>>>> a
>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>>>>> different?
>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
>>>>>> away?
>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
>>>>>> cut
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>> the
>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>> picked
>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>> the
>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
>> <
>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>>>>> Variable
>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>> NOTICE
>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
>>>>>> can
>>>>>>>>>>>>> not
>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
>>>>>> task
>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>> task
>>>>>>>>>>>>> causes
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>> push
>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>>>>> added to
>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
>>>>>> into
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>>>>> please
>>>>>>>>>>>>> set
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>>>>>> cluster,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>> If
>>>>>> you
>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>>>>> branch
>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.

Hi Bolke,

This one is a blocker: https://issues.apache.org/jira/browse/AIRFLOW-1731.

This requirers that we are sure that the /plugin/ folder is on the path:
https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py

And maybe add an additional test and updating the docs a bit. Hopefully
somewhere in the next few days, however I'm a bit busy with the Spark
Summit.

Cheers, Fokko

2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:

> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>
> What are the issues that are open?
>
> Cheers
> Bolke
>
> > On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org> wrote:
> >
> > K, I will plan on cutting an alpha1 later this week. Something for you
> guys to play with.
> >
> > On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
> <ma...@gmail.com>> wrote:
> > hi Chris,
> >
> > We are still running pre alpha. I am a bit preoccupied with preparing
> for a conference and the team is readying a release of one of our core
> products. So it will probably will be after this week when I get my hands
> dirty again.
> >
> > Cheers
> > Bolke
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
> <ma...@apache.org>> het volgende geschreven:
> > >
> > > Hey all,
> > >
> > > Checking in. Has anyone tested this in their environments? Stable
> releases
> > > depend on community involvement.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> criccomini@apache.org <ma...@apache.org>>
> > > wrote:
> > >
> > >> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
> but
> > >> will be included in alpha1.
> > >>
> > >>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid>
> wrote:
> > >>>
> > >>> Hi Chris,
> > >>>
> > >>> I know it's annoying to have last minute commit com in, but this is a
> > >>> highly desirable feature for folks using GCP operators, is it
> possible to
> > >>> include AIRFLOW-1635
> > >>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> > >>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> > >>> in?
> > >>> More than happy to offer help if there's something I can do.
> > >>> Thanks a lot.
> > >>>
> > >>> Feng
> > >>>
> > >>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> criccomini@apache.org <ma...@apache.org>>
> > >>> wrote:
> > >>>
> > >>>> Hey all,
> > >>>>
> > >>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
> here:
> > >>>>
> > >>>>  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> airflow/1.9.0alpha0/>
> > >>>>
> > >>>> The bin tarball can be installed with pip:
> > >>>>
> > >>>>  pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> > >>>>
> > >>>> The goal is to have the community install and run this to expose any
> > >>> bugs
> > >>>> before we move on to official release candidates.
> > >>>>
> > >>>> Here are the outstanding blocker bugs for 1.9.0:
> > >>>>
> > >>>>  AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >>>>  AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > >>> marked
> > >>>> as
> > >>>>  AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> exception
> > >>>> for
> > >>>> @on
> > >>>>  AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> > >>> stdout
> > >>>>  AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> > >>> for
> > >>>> @once
> > >>>>  AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> > >>>>
> > >>>> Cheers,
> > >>>> Chris
> > >>>>
> > >>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> criccomini@apache.org <ma...@apache.org>
> > >>>>
> > >>>> wrote:
> > >>>>
> > >>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> > >>>>>
> > >>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> > >>> criccomini@apache.org <ma...@apache.org>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Works for me. Will try and cut a beta before end of week.
> > >>>>>>
> > >>>>>> Blockers for 1.9.0 are:
> > >>>>>>
> > >>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > >>>> marked
> > >>>>>> as
> > >>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> > >>> fail
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Chris
> > >>>>>>>
> > >>>>>>> Can I suggest releasing a beta? The stable branch is only cut at
> RC
> > >>>>>>> time. Betas allow us a broader exposure. It also gives us a
> point of
> > >>>>>>> reference.
> > >>>>>>>
> > >>>>>>> In addition the list below are mostly longer standing issues that
> > >>> are
> > >>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
> 1258,
> > >>>> and
> > >>>>>>> 976 as blocker?
> > >>>>>>>
> > >>>>>>> Cheers
> > >>>>>>> Bolke
> > >>>>>>>
> > >>>>>>> Verstuurd vanaf mijn iPad
> > >>>>>>>
> > >>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> > >>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>> het volgende geschreven:
> > >>>>>>>>
> > >>>>>>>> Hey all,
> > >>>>>>>>
> > >>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> > >>> release,
> > >>>> but
> > >>>>>>>> seeing as there are several outstanding bugs, I'm going to
> delay.
> > >>>> Here
> > >>>>>>> are
> > >>>>>>>> the bugs that I'm tracking:
> > >>>>>>>>
> > >>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> are
> > >>>>>>> marked as
> > >>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> > >>>> exception
> > >>>>>>> for
> > >>>>>>>> @on
> > >>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
> to
> > >>>>>>> stdout
> > >>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> > >>> exception
> > >>>>>>> for @once
> > >>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
> > >>> Email
> > >>>> is
> > >>>>>>> Not
> > >>>>>>>> be
> > >>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> to
> > >>>> fail
> > >>>>>>>>
> > >>>>>>>> These are the priority issues. Once they're merged, I'll cut the
> > >>>>>>>> v1-9-stable and beta release.
> > >>>>>>>>
> > >>>>>>>> If you can help clean this up, that would be really appreciated.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> Chris
> > >>>>>>>>
> > >>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> > >>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Marked it for 1.9.0.
> > >>>>>>>>>
> > >>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> > >>> cjones@simpli.fi <ma...@simpli.fi>>
> > >>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> > >>>>>>> callbacks
> > >>>>>>>>>> are not working correctly without emails... Its not a major
> bug,
> > >>>> but
> > >>>>>>> it
> > >>>>>>>>>> does cause us some annoyance in our current deployment.
> > >>>>>>>>>>
> > >>>>>>>>>> Link to Jira:
> > >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> https://issues.apache.org/jira/browse/AIRFLOW-988>
> > >>>>>>>>>>
> > >>>>>>>>>> Link to PR:
> > >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
> https://github.com/apache/incubator-airflow/pull/2415>
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks!
> > >>>>>>>>>> Charlie Jones
> > >>>>>>>>>>
> > >>>>>>>>>> CHARLIE JONES
> > >>>>>>>>>> Data Engineer
> > >>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M:
> 972.821.7631 <tel:972.821.7631>
> > >>>>>>>>>> __________________________________________________
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Programmatic Performance.* Localized.*
> > >>>>>>>>>> __________________________________________________
> > >>>>>>>>>>
> > >>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> > >>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> http://www.simpli.fi/>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> > >>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Merged.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> > >>>>>>>>>> ryan.buckley@bluecore.com <ma...@bluecore.com>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> > >>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
> > >>> branch
> > >>>>>>> due
> > >>>>>>>>>> to
> > >>>>>>>>>>>> this issue.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
> https://github.com/apache/incubator-airflow/pull/2590>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Ryan
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> > >>>>>>>>>> <fokko@driesprong.frl
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
> https://github.com/apache/incubator-airflow/pull/2631>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Currently importing a custom logging configuration is not
> > >>> work
> > >>>> (as
> > >>>>>>>>>> far
> > >>>>>>>>>>>> as I
> > >>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Cheers, Fokko
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> > >>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>> :
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Done!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > >>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> michael.crawford@modernizingmedicine.com>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> > >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> > >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> https://github.com/apache/incubator-airflow/pull/2626> <
> > >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> https://github.com/apache/incubator-airflow/pull/2626>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
> > >>> edit
> > >>>>>>>>>> for
> > >>>>>>>>>>>> these
> > >>>>>>>>>>>>>>> types.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>> Mike
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
> forward
> > >>> to
> > >>>>>>>>>>> these
> > >>>>>>>>>>>>>> fixes!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > >>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
> > >>>>>>>>>>>>> .
> > >>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Can we get this in?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
> https://github.com/apache/incubator-airflow/commit/>
> > >>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> > >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
> https://github.com/apache/incubator-airflow/commit/>
> > >>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing
> to
> > >>>>>>>>>> have
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> 1.9.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> > >>>>>>>>>> Initial
> > >>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
> into
> > >>>>>>>>>> 1.9.0
> > >>>>>>>>>>> at
> > >>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> point.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > >>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> particular
> > >>>>>>>>>> fixed
> > >>>>>>>>>>>>> point
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
> than
> > >>> to
> > >>>> a
> > >>>>>>>>>>> git
> > >>>>>>>>>>>>>> pull.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Bolke
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> > >>> different?
> > >>>>>>>>>>> IIRC,
> > >>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > >>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> > >>> away?
> > >>>>>>>>>>>> Isn’t a
> > >>>>>>>>>>>>>>>>> beta
> > >>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>> bit smarter?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> - Bolke
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > >>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
> > >>> cut
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> stable
> > >>>>>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
> the
> > >>>>>>>>>> stable
> > >>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> cut, I
> > >>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
> picked
> > >>>> into
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> branch,
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
> the
> > >>>>>>>>>> release
> > >>>>>>>>>>>>> out.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
> <
> > >>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> > >>>>>>>>>> outstanding
> > >>>>>>>>>>> PRs
> > >>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> > >>> Variable
> > >>>>>>>>>>>>> endpoint
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> > >>> Airflow
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> > >>>>>>>>>> loggers
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
> NOTICE
> > >>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> > >>>>>>>>>> unneeded
> > >>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> > >>> HDFS
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> > >>> |airflow/jobs.py:create_dag_ru
> > >>>>>>>>>> n()
> > >>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> @on
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> > >>>>>>>>>> include
> > >>>>>>>>>>>>> paused
> > >>>>>>>>>>>>>>>>>> DAGs
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
> > >>> can
> > >>>>>>>>>> not
> > >>>>>>>>>>> log
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> stdout
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> > >>> task
> > >>>>>>>>>>>>> instances
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> > >>> |airflow/jobs.py:manage_slas()
> > >>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> @once
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
> task
> > >>>>>>>>>> causes
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> fail
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> > >>>>>>>>>>>>> BackfillJobTest.test_backfill_
> > >>>>>>>>>>>>>>>>>>>>> examples
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> > >>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> > >>>>>>>>>>>>>>>>> job
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> real
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> > >>>>>>>>>> matrix
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
> push
> > >>>>>>>>>> XComs
> > >>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
> XComs
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> > >>> added to
> > >>>>>>>>>>> hive
> > >>>>>>>>>>>>> conf
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
> > >>> into
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> 1.9.0
> > >>>>>>>>>>>>>>>>>>> branch
> > >>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> > >>> please
> > >>>>>>>>>> set
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> fix
> > >>>>>>>>>>>>>>>>>>>>> version
> > >>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> > >>>> cluster,
> > >>>>>>>>>>> and
> > >>>>>>>>>>>> it
> > >>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
> If
> > >>> you
> > >>>>>>>>>> run
> > >>>>>>>>>>>>>>>>> Airflow,
> > >>>>>>>>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> > >>> branch
> > >>>>>>>>>>>>> somewhere,
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> verify
> > >>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>> Chris
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> >
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

Im planning to upgrade our production to 1.9.0alphaX tomorrow.

What are the issues that are open?

Cheers
Bolke

> On 9 Oct 2017, at 20:24, Chris Riccomini <cr...@apache.org> wrote:
> 
> K, I will plan on cutting an alpha1 later this week. Something for you guys to play with.
> 
> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
> hi Chris,
> 
> We are still running pre alpha. I am a bit preoccupied with preparing for a conference and the team is readying a release of one of our core products. So it will probably will be after this week when I get my hands dirty again.
> 
> Cheers
> Bolke
> 
> Verstuurd vanaf mijn iPad
> 
> > Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org <ma...@apache.org>> het volgende geschreven:
> >
> > Hey all,
> >
> > Checking in. Has anyone tested this in their environments? Stable releases
> > depend on community involvement.
> >
> > Cheers,
> > Chris
> >
> > On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <criccomini@apache.org <ma...@apache.org>>
> > wrote:
> >
> >> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
> >> will be included in alpha1.
> >>
> >>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> I know it's annoying to have last minute commit com in, but this is a
> >>> highly desirable feature for folks using GCP operators, is it possible to
> >>> include AIRFLOW-1635
> >>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>> in?
> >>> More than happy to offer help if there's something I can do.
> >>> Thanks a lot.
> >>>
> >>> Feng
> >>>
> >>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <criccomini@apache.org <ma...@apache.org>>
> >>> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
> >>>>
> >>>>  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/>
> >>>>
> >>>> The bin tarball can be installed with pip:
> >>>>
> >>>>  pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>
> >>>> The goal is to have the community install and run this to expose any
> >>> bugs
> >>>> before we move on to official release candidates.
> >>>>
> >>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>
> >>>>  AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>  AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>> marked
> >>>> as
> >>>>  AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> >>>> for
> >>>> @on
> >>>>  AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>> stdout
> >>>>  AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> >>> for
> >>>> @once
> >>>>  AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <criccomini@apache.org <ma...@apache.org>
> >>>>
> >>>> wrote:
> >>>>
> >>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>
> >>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>> criccomini@apache.org <ma...@apache.org>>
> >>>>> wrote:
> >>>>>
> >>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>
> >>>>>> Blockers for 1.9.0 are:
> >>>>>>
> >>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>>> marked
> >>>>>> as
> >>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >>> fail
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Chris
> >>>>>>>
> >>>>>>> Can I suggest releasing a beta? The stable branch is only cut at RC
> >>>>>>> time. Betas allow us a broader exposure. It also gives us a point of
> >>>>>>> reference.
> >>>>>>>
> >>>>>>> In addition the list below are mostly longer standing issues that
> >>> are
> >>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
> >>>> and
> >>>>>>> 976 as blocker?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Bolke
> >>>>>>>
> >>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>
> >>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>> het volgende geschreven:
> >>>>>>>>
> >>>>>>>> Hey all,
> >>>>>>>>
> >>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>> release,
> >>>> but
> >>>>>>>> seeing as there are several outstanding bugs, I'm going to delay.
> >>>> Here
> >>>>>>> are
> >>>>>>>> the bugs that I'm tracking:
> >>>>>>>>
> >>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>>>>>> marked as
> >>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>> exception
> >>>>>>> for
> >>>>>>>> @on
> >>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>>>>>> stdout
> >>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>> exception
> >>>>>>> for @once
> >>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
> >>> Email
> >>>> is
> >>>>>>> Not
> >>>>>>>> be
> >>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >>>> fail
> >>>>>>>>
> >>>>>>>> These are the priority issues. Once they're merged, I'll cut the
> >>>>>>>> v1-9-stable and beta release.
> >>>>>>>>
> >>>>>>>> If you can help clean this up, that would be really appreciated.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>
> >>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>> cjones@simpli.fi <ma...@simpli.fi>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> >>>>>>> callbacks
> >>>>>>>>>> are not working correctly without emails... Its not a major bug,
> >>>> but
> >>>>>>> it
> >>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>
> >>>>>>>>>> Link to Jira:
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>
> >>>>>>>>>> Link to PR:
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>> Charlie Jones
> >>>>>>>>>>
> >>>>>>>>>> CHARLIE JONES
> >>>>>>>>>> Data Engineer
> >>>>>>>>>> cjones@simpli.fi <ma...@simpli.fi>  |  M: 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>> __________________________________________________
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>> __________________________________________________
> >>>>>>>>>>
> >>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <http://www.simpli.fi/>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Merged.
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>> ryan.buckley@bluecore.com <ma...@bluecore.com>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
> >>> branch
> >>>>>>> due
> >>>>>>>>>> to
> >>>>>>>>>>>> this issue.
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Ryan
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Currently importing a custom logging configuration is not
> >>> work
> >>>> (as
> >>>>>>>>>> far
> >>>>>>>>>>>> as I
> >>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>> :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <ma...@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
> >>> edit
> >>>>>>>>>> for
> >>>>>>>>>>>> these
> >>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
> >>> to
> >>>>>>>>>>> these
> >>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>> alex.guziel@airbnb.com <ma...@airbnb.com>
> >>>>>>>>>>>>> .
> >>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
> >>>>>>>>>> have
> >>>>>>>>>>> in
> >>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> >>>>>>>>>> Initial
> >>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get into
> >>>>>>>>>> 1.9.0
> >>>>>>>>>>> at
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
> >>>>>>>>>> fixed
> >>>>>>>>>>>>> point
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
> >>> to
> >>>> a
> >>>>>>>>>>> git
> >>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>> different?
> >>>>>>>>>>> IIRC,
> >>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> >>>>>>>>>>>>>> bdbruin@gmail.com <ma...@gmail.com>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> >>> away?
> >>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
> >>> cut
> >>>>>>>>>> the
> >>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> >>>>>>>>>> stable
> >>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry picked
> >>>> into
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> >>>>>>>>>> release
> >>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>> criccomini@apache.org <ma...@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>>>>>>>>> outstanding
> >>>>>>>>>>> PRs
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> >>> Variable
> >>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> >>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> >>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> >>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> >>>>>>>>>> unneeded
> >>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> >>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>> n()
> >>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> >>>>>>>>>> include
> >>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
> >>> can
> >>>>>>>>>> not
> >>>>>>>>>>> log
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> >>> task
> >>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> >>>>>>>>>> causes
> >>>>>>>>>>> it
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> >>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> >>>>>>>>>> XComs
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> >>> added to
> >>>>>>>>>>> hive
> >>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
> >>> into
> >>>>>>>>>> the
> >>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> >>> please
> >>>>>>>>>> set
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> >>>> cluster,
> >>>>>>>>>>> and
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
> >>> you
> >>>>>>>>>> run
> >>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> >>> branch
> >>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

K, I will plan on cutting an alpha1 later this week. Something for you guys
to play with.

On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> hi Chris,
>
> We are still running pre alpha. I am a bit preoccupied with preparing for
> a conference and the team is readying a release of one of our core
> products. So it will probably will be after this week when I get my hands
> dirty again.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <cr...@apache.org>
> het volgende geschreven:
> >
> > Hey all,
> >
> > Checking in. Has anyone tested this in their environments? Stable
> releases
> > depend on community involvement.
> >
> > Cheers,
> > Chris
> >
> > On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
> >> will be included in alpha1.
> >>
> >>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid>
> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> I know it's annoying to have last minute commit com in, but this is a
> >>> highly desirable feature for folks using GCP operators, is it possible
> to
> >>> include AIRFLOW-1635
> >>> <https://github.com/apache/incubator-airflow/commit/b3e985a3
> >>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>> in?
> >>> More than happy to offer help if there's something I can do.
> >>> Thanks a lot.
> >>>
> >>> Feng
> >>>
> >>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <criccomini@apache.org
> >
> >>> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
> >>>>
> >>>>  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0alpha0/
> >>>>
> >>>> The bin tarball can be installed with pip:
> >>>>
> >>>>  pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>
> >>>> The goal is to have the community install and run this to expose any
> >>> bugs
> >>>> before we move on to official release candidates.
> >>>>
> >>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>
> >>>>  AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>  AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>> marked
> >>>> as
> >>>>  AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> exception
> >>>> for
> >>>> @on
> >>>>  AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>> stdout
> >>>>  AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> >>> for
> >>>> @once
> >>>>  AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> criccomini@apache.org
> >>>>
> >>>> wrote:
> >>>>
> >>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>
> >>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>> criccomini@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>
> >>>>>> Blockers for 1.9.0 are:
> >>>>>>
> >>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>>> marked
> >>>>>> as
> >>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >>> fail
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Chris
> >>>>>>>
> >>>>>>> Can I suggest releasing a beta? The stable branch is only cut at RC
> >>>>>>> time. Betas allow us a broader exposure. It also gives us a point
> of
> >>>>>>> reference.
> >>>>>>>
> >>>>>>> In addition the list below are mostly longer standing issues that
> >>> are
> >>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
> 1258,
> >>>> and
> >>>>>>> 976 as blocker?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Bolke
> >>>>>>>
> >>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>
> >>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>> criccomini@apache.org>
> >>>>>>> het volgende geschreven:
> >>>>>>>>
> >>>>>>>> Hey all,
> >>>>>>>>
> >>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>> release,
> >>>> but
> >>>>>>>> seeing as there are several outstanding bugs, I'm going to delay.
> >>>> Here
> >>>>>>> are
> >>>>>>>> the bugs that I'm tracking:
> >>>>>>>>
> >>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>>>>>> marked as
> >>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>> exception
> >>>>>>> for
> >>>>>>>> @on
> >>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>>>>>> stdout
> >>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>> exception
> >>>>>>> for @once
> >>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
> >>> Email
> >>>> is
> >>>>>>> Not
> >>>>>>>> be
> >>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
> >>>> fail
> >>>>>>>>
> >>>>>>>> These are the priority issues. Once they're merged, I'll cut the
> >>>>>>>> v1-9-stable and beta release.
> >>>>>>>>
> >>>>>>>> If you can help clean this up, that would be really appreciated.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>> criccomini@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>
> >>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>> cjones@simpli.fi>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> >>>>>>> callbacks
> >>>>>>>>>> are not working correctly without emails... Its not a major bug,
> >>>> but
> >>>>>>> it
> >>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>
> >>>>>>>>>> Link to Jira:
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988
> >>>>>>>>>>
> >>>>>>>>>> Link to PR:
> >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>> Charlie Jones
> >>>>>>>>>>
> >>>>>>>>>> CHARLIE JONES
> >>>>>>>>>> Data Engineer
> >>>>>>>>>> cjones@simpli.fi  |  M: 972.821.7631
> >>>>>>>>>> __________________________________________________
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>> __________________________________________________
> >>>>>>>>>>
> >>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>> 800.840.0768  |  www.simpli.fi
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Merged.
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>> ryan.buckley@bluecore.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
> >>> branch
> >>>>>>> due
> >>>>>>>>>> to
> >>>>>>>>>>>> this issue.
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Ryan
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Currently importing a custom logging configuration is not
> >>> work
> >>>> (as
> >>>>>>>>>> far
> >>>>>>>>>>>> as I
> >>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>> criccomini@apache.org
> >>>>>>>> :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
> >>> edit
> >>>>>>>>>> for
> >>>>>>>>>>>> these
> >>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
> >>> to
> >>>>>>>>>>> these
> >>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>> alex.guziel@airbnb.com
> >>>>>>>>>>>>> .
> >>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
> >>>>>>>>>> have
> >>>>>>>>>>> in
> >>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> >>>>>>>>>> Initial
> >>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get into
> >>>>>>>>>> 1.9.0
> >>>>>>>>>>> at
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>>>>>>>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
> >>>>>>>>>> fixed
> >>>>>>>>>>>>> point
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
> >>> to
> >>>> a
> >>>>>>>>>>> git
> >>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>> different?
> >>>>>>>>>>> IIRC,
> >>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> >>>>>>>>>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> >>> away?
> >>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
> >>> cut
> >>>>>>>>>> the
> >>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> >>>>>>>>>> stable
> >>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry picked
> >>>> into
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> >>>>>>>>>> release
> >>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>>>>>>>>> outstanding
> >>>>>>>>>>> PRs
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> >>> Variable
> >>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> >>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> >>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> >>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> >>>>>>>>>> unneeded
> >>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> >>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>> n()
> >>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> >>>>>>>>>> include
> >>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
> >>> can
> >>>>>>>>>> not
> >>>>>>>>>>> log
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> >>> task
> >>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> >>>>>>>>>> causes
> >>>>>>>>>>> it
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> >>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> >>>>>>>>>> XComs
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
> >>> added to
> >>>>>>>>>>> hive
> >>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
> >>> into
> >>>>>>>>>> the
> >>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> >>> please
> >>>>>>>>>> set
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> >>>> cluster,
> >>>>>>>>>>> and
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
> >>> you
> >>>>>>>>>> run
> >>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> >>> branch
> >>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

hi Chris,

We are still running pre alpha. I am a bit preoccupied with preparing for a conference and the team is readying a release of one of our core products. So it will probably will be after this week when I get my hands dirty again. 

Cheers
Bolke

Verstuurd vanaf mijn iPad

> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <cr...@apache.org> het volgende geschreven:
> 
> Hey all,
> 
> Checking in. Has anyone tested this in their environments? Stable releases
> depend on community involvement.
> 
> Cheers,
> Chris
> 
> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <cr...@apache.org>
> wrote:
> 
>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
>> will be included in alpha1.
>> 
>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid> wrote:
>>> 
>>> Hi Chris,
>>> 
>>> I know it's annoying to have last minute commit com in, but this is a
>>> highly desirable feature for folks using GCP operators, is it possible to
>>> include AIRFLOW-1635
>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3
>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>> in?
>>> More than happy to offer help if there's something I can do.
>>> Thanks a lot.
>>> 
>>> Feng
>>> 
>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>> 
>>>> Hey all,
>>>> 
>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
>>>> 
>>>>  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/
>>>> 
>>>> The bin tarball can be installed with pip:
>>>> 
>>>>  pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>> 
>>>> The goal is to have the community install and run this to expose any
>>> bugs
>>>> before we move on to official release candidates.
>>>> 
>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>> 
>>>>  AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>  AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> marked
>>>> as
>>>>  AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
>>>> for
>>>> @on
>>>>  AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>> stdout
>>>>  AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>> for
>>>> @once
>>>>  AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <criccomini@apache.org
>>>> 
>>>> wrote:
>>>> 
>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>>>> 
>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>> criccomini@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>> 
>>>>>> Blockers for 1.9.0 are:
>>>>>> 
>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>> marked
>>>>>> as
>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>> fail
>>>>>> 
>>>>>> 
>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi Chris
>>>>>>> 
>>>>>>> Can I suggest releasing a beta? The stable branch is only cut at RC
>>>>>>> time. Betas allow us a broader exposure. It also gives us a point of
>>>>>>> reference.
>>>>>>> 
>>>>>>> In addition the list below are mostly longer standing issues that
>>> are
>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
>>>> and
>>>>>>> 976 as blocker?
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>> 
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>> 
>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>>> criccomini@apache.org>
>>>>>>> het volgende geschreven:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>> release,
>>>> but
>>>>>>>> seeing as there are several outstanding bugs, I'm going to delay.
>>>> Here
>>>>>>> are
>>>>>>>> the bugs that I'm tracking:
>>>>>>>> 
>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>>>>>> marked as
>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>> exception
>>>>>>> for
>>>>>>>> @on
>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>>>>>> stdout
>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>> exception
>>>>>>> for @once
>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>> Email
>>>> is
>>>>>>> Not
>>>>>>>> be
>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>>> fail
>>>>>>>> 
>>>>>>>> These are the priority issues. Once they're merged, I'll cut the
>>>>>>>> v1-9-stable and beta release.
>>>>>>>> 
>>>>>>>> If you can help clean this up, that would be really appreciated.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>>>>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>> 
>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>> cjones@simpli.fi>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>>>>>>> callbacks
>>>>>>>>>> are not working correctly without emails... Its not a major bug,
>>>> but
>>>>>>> it
>>>>>>>>>> does cause us some annoyance in our current deployment.
>>>>>>>>>> 
>>>>>>>>>> Link to Jira:
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>>>>>>>>> 
>>>>>>>>>> Link to PR:
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> Charlie Jones
>>>>>>>>>> 
>>>>>>>>>> CHARLIE JONES
>>>>>>>>>> Data Engineer
>>>>>>>>>> cjones@simpli.fi  |  M: 972.821.7631
>>>>>>>>>> __________________________________________________
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>> __________________________________________________
>>>>>>>>>> 
>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>>>>>>>>> 800.840.0768  |  www.simpli.fi
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>>>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Merged.
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>>>>>>>>> ryan.buckley@bluecore.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>> branch
>>>>>>> due
>>>>>>>>>> to
>>>>>>>>>>>> this issue.
>>>>>>>>>>>> 
>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Ryan
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>> work
>>>> (as
>>>>>>>>>> far
>>>>>>>>>>>> as I
>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>>> criccomini@apache.org
>>>>>>>> :
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
>>> edit
>>>>>>>>>> for
>>>>>>>>>>>> these
>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
>>> to
>>>>>>>>>>> these
>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>> alex.guziel@airbnb.com
>>>>>>>>>>>>> .
>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Can we get this in?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
>>>>>>>>>> have
>>>>>>>>>>> in
>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>>>>>>>>> Initial
>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get into
>>>>>>>>>> 1.9.0
>>>>>>>>>>> at
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
>>>>>>>>>> fixed
>>>>>>>>>>>>> point
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
>>> to
>>>> a
>>>>>>>>>>> git
>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>> different?
>>>>>>>>>>> IIRC,
>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
>>> away?
>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
>>> cut
>>>>>>>>>> the
>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
>>>>>>>>>> stable
>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry picked
>>>> into
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get the
>>>>>>>>>> release
>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>>>>>>>>> outstanding
>>>>>>>>>>> PRs
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>> Variable
>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>>>>>>>>> unneeded
>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>> n()
>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>>>>>>>>> include
>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
>>> can
>>>>>>>>>> not
>>>>>>>>>>> log
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
>>> task
>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>>>>>>>>>> causes
>>>>>>>>>>> it
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>>>>>>>>>> XComs
>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>> added to
>>>>>>>>>>> hive
>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
>>> into
>>>>>>>>>> the
>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>> please
>>>>>>>>>> set
>>>>>>>>>>>> the
>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>>> cluster,
>>>>>>>>>>> and
>>>>>>>>>>>> it
>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
>>> you
>>>>>>>>>> run
>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>> branch
>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

Checking in. Has anyone tested this in their environments? Stable releases
depend on community involvement.

Cheers,
Chris

On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <cr...@apache.org>
wrote:

> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
> will be included in alpha1.
>
> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid> wrote:
>
>> Hi Chris,
>>
>> I know it's annoying to have last minute commit com in, but this is a
>> highly desirable feature for folks using GCP operators, is it possible to
>> include AIRFLOW-1635
>> <https://github.com/apache/incubator-airflow/commit/b3e985a3
>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>> in?
>> More than happy to offer help if there's something I can do.
>> Thanks a lot.
>>
>> Feng
>>
>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>> > Hey all,
>> >
>> > I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
>> >
>> >   https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/
>> >
>> > The bin tarball can be installed with pip:
>> >
>> >   pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>> >
>> > The goal is to have the community install and run this to expose any
>> bugs
>> > before we move on to official release candidates.
>> >
>> > Here are the outstanding blocker bugs for 1.9.0:
>> >
>> >   AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> >   AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> marked
>> > as
>> >   AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
>> > for
>> > @on
>> >   AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>> stdout
>> >   AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>> for
>> > @once
>> >   AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>> >
>> > Cheers,
>> > Chris
>> >
>> > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <criccomini@apache.org
>> >
>> > wrote:
>> >
>> > > Welp. Work got in the way, so I'll cut the beta on Monday. :)
>> > >
>> > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>> criccomini@apache.org>
>> > > wrote:
>> > >
>> > >> Works for me. Will try and cut a beta before end of week.
>> > >>
>> > >> Blockers for 1.9.0 are:
>> > >>
>> > >> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> > >> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> > marked
>> > >> as
>> > >> AIRFLOW-976  |Bug         |Mark success running task causes it to
>> fail
>> > >>
>> > >>
>> > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> Hi Chris
>> > >>>
>> > >>> Can I suggest releasing a beta? The stable branch is only cut at RC
>> > >>> time. Betas allow us a broader exposure. It also gives us a point of
>> > >>> reference.
>> > >>>
>> > >>> In addition the list below are mostly longer standing issues that
>> are
>> > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
>> > and
>> > >>> 976 as blocker?
>> > >>>
>> > >>> Cheers
>> > >>> Bolke
>> > >>>
>> > >>> Verstuurd vanaf mijn iPad
>> > >>>
>> > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>> > criccomini@apache.org>
>> > >>> het volgende geschreven:
>> > >>> >
>> > >>> > Hey all,
>> > >>> >
>> > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>> release,
>> > but
>> > >>> > seeing as there are several outstanding bugs, I'm going to delay.
>> > Here
>> > >>> are
>> > >>> > the bugs that I'm tracking:
>> > >>> >
>> > >>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> > >>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> > >>> marked as
>> > >>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>> > exception
>> > >>> for
>> > >>> > @on
>> > >>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>> > >>> stdout
>> > >>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>> exception
>> > >>> for @once
>> > >>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>> Email
>> > is
>> > >>> Not
>> > >>> > be
>> > >>> > AIRFLOW-976  |Bug         |Mark success running task causes it to
>> > fail
>> > >>> >
>> > >>> > These are the priority issues. Once they're merged, I'll cut the
>> > >>> > v1-9-stable and beta release.
>> > >>> >
>> > >>> > If you can help clean this up, that would be really appreciated.
>> > >>> >
>> > >>> > Cheers,
>> > >>> > Chris
>> > >>> >
>> > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>> > >>> criccomini@apache.org>
>> > >>> > wrote:
>> > >>> >
>> > >>> >> Marked it for 1.9.0.
>> > >>> >>
>> > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>> cjones@simpli.fi>
>> > >>> wrote:
>> > >>> >>>
>> > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>> > >>> callbacks
>> > >>> >>> are not working correctly without emails... Its not a major bug,
>> > but
>> > >>> it
>> > >>> >>> does cause us some annoyance in our current deployment.
>> > >>> >>>
>> > >>> >>> Link to Jira:
>> > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
>> > >>> >>>
>> > >>> >>> Link to PR:
>> > >>> >>> https://github.com/apache/incubator-airflow/pull/2415
>> > >>> >>>
>> > >>> >>> Thanks!
>> > >>> >>> Charlie Jones
>> > >>> >>>
>> > >>> >>> CHARLIE JONES
>> > >>> >>> Data Engineer
>> > >>> >>> cjones@simpli.fi  |  M: 972.821.7631
>> > >>> >>> __________________________________________________
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> Programmatic Performance.* Localized.*
>> > >>> >>> __________________________________________________
>> > >>> >>>
>> > >>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> > >>> >>> 800.840.0768  |  www.simpli.fi
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>> > >>> criccomini@apache.org>
>> > >>> >>> wrote:
>> > >>> >>>
>> > >>> >>>> Merged.
>> > >>> >>>>
>> > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> > >>> >>> ryan.buckley@bluecore.com>
>> > >>> >>>> wrote:
>> > >>> >>>>
>> > >>> >>>>> Would it be possible to include AIRFLOW-1587?
>> > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0
>> branch
>> > >>> due
>> > >>> >>> to
>> > >>> >>>>> this issue.
>> > >>> >>>>>
>> > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
>> > >>> >>>>>
>> > >>> >>>>> Thanks,
>> > >>> >>>>> Ryan
>> > >>> >>>>>
>> > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> > >>> >>> <fokko@driesprong.frl
>> > >>> >>>>>
>> > >>> >>>>> wrote:
>> > >>> >>>>>
>> > >>> >>>>>> Hi All,
>> > >>> >>>>>>
>> > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>> > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
>> > >>> >>>>>>
>> > >>> >>>>>> Currently importing a custom logging configuration is not
>> work
>> > (as
>> > >>> >>> far
>> > >>> >>>>> as I
>> > >>> >>>>>> know). Any feedback on the PR would also be appreciated.
>> > >>> >>>>>>
>> > >>> >>>>>> Cheers, Fokko
>> > >>> >>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>> > criccomini@apache.org
>> > >>> >:
>> > >>> >>>>>>
>> > >>> >>>>>>> Done!
>> > >>> >>>>>>>
>> > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> > >>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>> > >>> >>>>>>>
>> > >>> >>>>>>>> Can you slide the aws and emr connection type fix in?
>> > >>> >>>>>>>>
>> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>> > >>> >>>>>>>>
>> > >>> >>>>>>>> It keeps the connection type from getting blanked out on
>> edit
>> > >>> >>> for
>> > >>> >>>>> these
>> > >>> >>>>>>>> types.
>> > >>> >>>>>>>>
>> > >>> >>>>>>>> Thanks,
>> > >>> >>>>>>>> Mike
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>> > >>> >>>>> criccomini@apache.org>
>> > >>> >>>>>>>> wrote:
>> > >>> >>>>>>>>>
>> > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
>> to
>> > >>> >>>> these
>> > >>> >>>>>>> fixes!
>> > >>> >>>>>>>>>
>> > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>> > >>> >>>>> alex.guziel@airbnb.com
>> > >>> >>>>>> .
>> > >>> >>>>>>>> invalid
>> > >>> >>>>>>>>>> wrote:
>> > >>> >>>>>>>>>
>> > >>> >>>>>>>>>> Can we get this in?
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
>> > >>> >>> have
>> > >>> >>>> in
>> > >>> >>>>>> 1.9.
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>> > >>> >>>>>>>> criccomini@apache.org>
>> > >>> >>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>> > >>> >>> Initial
>> > >>> >>>>>>> warning
>> > >>> >>>>>>>>>>> stands, that I will start locking down what can get into
>> > >>> >>> 1.9.0
>> > >>> >>>> at
>> > >>> >>>>>>> that
>> > >>> >>>>>>>>>>> point.
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>> > >>> >>>>>> bdbruin@gmail.com>
>> > >>> >>>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
>> > >>> >>> fixed
>> > >>> >>>>>> point
>> > >>> >>>>>>> in
>> > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
>> to
>> > a
>> > >>> >>>> git
>> > >>> >>>>>>> pull.
>> > >>> >>>>>>>>>>>>
>> > >>> >>>>>>>>>>>> Bolke
>> > >>> >>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>> > >>> >>>>> criccomini@apache.org
>> > >>> >>>>>>>
>> > >>> >>>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly
>> different?
>> > >>> >>>> IIRC,
>> > >>> >>>>>> it's
>> > >>> >>>>>>>>>>>>> basically the same, just no vote, right?
>> > >>> >>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>> > >>> >>>>>>> bdbruin@gmail.com>
>> > >>> >>>>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
>> away?
>> > >>> >>>>> Isn’t a
>> > >>> >>>>>>>>>> beta
>> > >>> >>>>>>>>>>> a
>> > >>> >>>>>>>>>>>>>> bit smarter?
>> > >>> >>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>> - Bolke
>> > >>> >>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>> > >>> >>>>>> criccomini@apache.org
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> Hey all,
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
>> cut
>> > >>> >>> the
>> > >>> >>>>>> stable
>> > >>> >>>>>>>>>>> branch
>> > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
>> > >>> >>> stable
>> > >>> >>>>>> branch
>> > >>> >>>>>>>>>> is
>> > >>> >>>>>>>>>>>>>> cut, I
>> > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked
>> > into
>> > >>> >>>> the
>> > >>> >>>>>>>>>> branch,
>> > >>> >>>>>>>>>>>> and
>> > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
>> > >>> >>> release
>> > >>> >>>>>> out.
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> Cheers,
>> > >>> >>>>>>>>>>>>>>> Chris
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>> > >>> >>>>>>>>>>>> criccomini@apache.org
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>> wrote:
>> > >>> >>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> Hey all,
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>> > >>> >>> outstanding
>> > >>> >>>> PRs
>> > >>> >>>>>>> that
>> > >>> >>>>>>>>>>> are
>> > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>> Variable
>> > >>> >>>>>> endpoint
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>> Airflow
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>> > >>> >>> loggers
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>> > >>> >>> issue
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>> > >>> >>> unneeded
>> > >>> >>>>> code
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
>> HDFS
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>> |airflow/jobs.py:create_dag_ru
>> > >>> >>> n()
>> > >>> >>>>>>>>>> exception
>> > >>> >>>>>>>>>>>> for
>> > >>> >>>>>>>>>>>>>>>> @on
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>> > >>> >>> include
>> > >>> >>>>>> paused
>> > >>> >>>>>>>>>>> DAGs
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
>> can
>> > >>> >>> not
>> > >>> >>>> log
>> > >>> >>>>>> to
>> > >>> >>>>>>>>>>> stdout
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
>> task
>> > >>> >>>>>> instances
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>> |airflow/jobs.py:manage_slas()
>> > >>> >>>>>> exception
>> > >>> >>>>>>>>>> for
>> > >>> >>>>>>>>>>>>>>>> @once
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>> > >>> >>> causes
>> > >>> >>>> it
>> > >>> >>>>>> to
>> > >>> >>>>>>>>>> fail
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>> > >>> >>>>>> BackfillJobTest.test_backfill_
>> > >>> >>>>>>>>>>>>>> examples
>> > >>> >>>>>>>>>>>>>>>> to
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>> > >>> >>>>>> tests.CoreTest.test_scheduler_
>> > >>> >>>>>>>>>> job
>> > >>> >>>>>>>>>>>> to
>> > >>> >>>>>>>>>>>>>>>> real
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>> > >>> >>> matrix
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>> > >>> >>> XComs
>> > >>> >>>> by
>> > >>> >>>>>>>>>> default
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>> > >>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>> added to
>> > >>> >>>> hive
>> > >>> >>>>>> conf
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
>> into
>> > >>> >>> the
>> > >>> >>>>>> 1.9.0
>> > >>> >>>>>>>>>>>> branch
>> > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>> please
>> > >>> >>> set
>> > >>> >>>>> the
>> > >>> >>>>>>> fix
>> > >>> >>>>>>>>>>>>>> version
>> > >>> >>>>>>>>>>>>>>>> to 1.9.0.
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>> > cluster,
>> > >>> >>>> and
>> > >>> >>>>> it
>> > >>> >>>>>>> has
>> > >>> >>>>>>>>>>>> been
>> > >>> >>>>>>>>>>>>>>>> running smoothly for several days.
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
>> you
>> > >>> >>> run
>> > >>> >>>>>>>>>> Airflow,
>> > >>> >>>>>>>>>>>> it's
>> > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>> branch
>> > >>> >>>>>> somewhere,
>> > >>> >>>>>>>>>> and
>> > >>> >>>>>>>>>>>>>> verify
>> > >>> >>>>>>>>>>>>>>>> it's working for your workload. **
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>>> Cheers,
>> > >>> >>>>>>>>>>>>>>>> Chris
>> > >>> >>>>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>
>> > >>> >>>>>>>>>>>>
>> > >>> >>>>>>>>>>>
>> > >>> >>>>>>>>>>
>> > >>> >>>>>>>>
>> > >>> >>>>>>>>
>> > >>> >>>>>>>
>> > >>> >>>>>>
>> > >>> >>>>>
>> > >>> >>>>
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>
>> > >>
>> > >
>> >
>>
>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but
will be included in alpha1.

On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fe...@google.com.invalid> wrote:

> Hi Chris,
>
> I know it's annoying to have last minute commit com in, but this is a
> highly desirable feature for folks using GCP operators, is it possible to
> include AIRFLOW-1635
> <https://github.com/apache/incubator-airflow/commit/
> b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4>
> in?
> More than happy to offer help if there's something I can do.
> Thanks a lot.
>
> Feng
>
> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey all,
> >
> > I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
> >
> >   https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/
> >
> > The bin tarball can be installed with pip:
> >
> >   pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >
> > The goal is to have the community install and run this to expose any bugs
> > before we move on to official release candidates.
> >
> > Here are the outstanding blocker bugs for 1.9.0:
> >
> >   AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >   AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> marked
> > as
> >   AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> > for
> > @on
> >   AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> stdout
> >   AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
> > @once
> >   AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >
> > Cheers,
> > Chris
> >
> > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> > > Welp. Work got in the way, so I'll cut the beta on Monday. :)
> > >
> > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> criccomini@apache.org>
> > > wrote:
> > >
> > >> Works for me. Will try and cut a beta before end of week.
> > >>
> > >> Blockers for 1.9.0 are:
> > >>
> > >> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > marked
> > >> as
> > >> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> > >>
> > >>
> > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Chris
> > >>>
> > >>> Can I suggest releasing a beta? The stable branch is only cut at RC
> > >>> time. Betas allow us a broader exposure. It also gives us a point of
> > >>> reference.
> > >>>
> > >>> In addition the list below are mostly longer standing issues that are
> > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
> > and
> > >>> 976 as blocker?
> > >>>
> > >>> Cheers
> > >>> Bolke
> > >>>
> > >>> Verstuurd vanaf mijn iPad
> > >>>
> > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> > criccomini@apache.org>
> > >>> het volgende geschreven:
> > >>> >
> > >>> > Hey all,
> > >>> >
> > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release,
> > but
> > >>> > seeing as there are several outstanding bugs, I'm going to delay.
> > Here
> > >>> are
> > >>> > the bugs that I'm tracking:
> > >>> >
> > >>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > >>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> > >>> marked as
> > >>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> > exception
> > >>> for
> > >>> > @on
> > >>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> > >>> stdout
> > >>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> > >>> for @once
> > >>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email
> > is
> > >>> Not
> > >>> > be
> > >>> > AIRFLOW-976  |Bug         |Mark success running task causes it to
> > fail
> > >>> >
> > >>> > These are the priority issues. Once they're merged, I'll cut the
> > >>> > v1-9-stable and beta release.
> > >>> >
> > >>> > If you can help clean this up, that would be really appreciated.
> > >>> >
> > >>> > Cheers,
> > >>> > Chris
> > >>> >
> > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> > >>> criccomini@apache.org>
> > >>> > wrote:
> > >>> >
> > >>> >> Marked it for 1.9.0.
> > >>> >>
> > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi
> >
> > >>> wrote:
> > >>> >>>
> > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> > >>> callbacks
> > >>> >>> are not working correctly without emails... Its not a major bug,
> > but
> > >>> it
> > >>> >>> does cause us some annoyance in our current deployment.
> > >>> >>>
> > >>> >>> Link to Jira:
> > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> > >>> >>>
> > >>> >>> Link to PR:
> > >>> >>> https://github.com/apache/incubator-airflow/pull/2415
> > >>> >>>
> > >>> >>> Thanks!
> > >>> >>> Charlie Jones
> > >>> >>>
> > >>> >>> CHARLIE JONES
> > >>> >>> Data Engineer
> > >>> >>> cjones@simpli.fi  |  M: 972.821.7631
> > >>> >>> __________________________________________________
> > >>> >>>
> > >>> >>>
> > >>> >>> Programmatic Performance.* Localized.*
> > >>> >>> __________________________________________________
> > >>> >>>
> > >>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> > >>> >>> 800.840.0768  |  www.simpli.fi
> > >>> >>>
> > >>> >>>
> > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> > >>> criccomini@apache.org>
> > >>> >>> wrote:
> > >>> >>>
> > >>> >>>> Merged.
> > >>> >>>>
> > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> > >>> >>> ryan.buckley@bluecore.com>
> > >>> >>>> wrote:
> > >>> >>>>
> > >>> >>>>> Would it be possible to include AIRFLOW-1587?
> > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0
> branch
> > >>> due
> > >>> >>> to
> > >>> >>>>> this issue.
> > >>> >>>>>
> > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> > >>> >>>>>
> > >>> >>>>> Thanks,
> > >>> >>>>> Ryan
> > >>> >>>>>
> > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> > >>> >>> <fokko@driesprong.frl
> > >>> >>>>>
> > >>> >>>>> wrote:
> > >>> >>>>>
> > >>> >>>>>> Hi All,
> > >>> >>>>>>
> > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> > >>> >>>>>>
> > >>> >>>>>> Currently importing a custom logging configuration is not work
> > (as
> > >>> >>> far
> > >>> >>>>> as I
> > >>> >>>>>> know). Any feedback on the PR would also be appreciated.
> > >>> >>>>>>
> > >>> >>>>>> Cheers, Fokko
> > >>> >>>>>>
> > >>> >>>>>>
> > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> > criccomini@apache.org
> > >>> >:
> > >>> >>>>>>
> > >>> >>>>>>> Done!
> > >>> >>>>>>>
> > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > >>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> > >>> >>>>>>>
> > >>> >>>>>>>> Can you slide the aws and emr connection type fix in?
> > >>> >>>>>>>>
> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> > >>> >>>>>>>>
> > >>> >>>>>>>> It keeps the connection type from getting blanked out on
> edit
> > >>> >>> for
> > >>> >>>>> these
> > >>> >>>>>>>> types.
> > >>> >>>>>>>>
> > >>> >>>>>>>> Thanks,
> > >>> >>>>>>>> Mike
> > >>> >>>>>>>>
> > >>> >>>>>>>>
> > >>> >>>>>>>>
> > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > >>> >>>>> criccomini@apache.org>
> > >>> >>>>>>>> wrote:
> > >>> >>>>>>>>>
> > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
> to
> > >>> >>>> these
> > >>> >>>>>>> fixes!
> > >>> >>>>>>>>>
> > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > >>> >>>>> alex.guziel@airbnb.com
> > >>> >>>>>> .
> > >>> >>>>>>>> invalid
> > >>> >>>>>>>>>> wrote:
> > >>> >>>>>>>>>
> > >>> >>>>>>>>>> Can we get this in?
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
> > >>> >>> have
> > >>> >>>> in
> > >>> >>>>>> 1.9.
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > >>> >>>>>>>> criccomini@apache.org>
> > >>> >>>>>>>>>> wrote:
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> > >>> >>> Initial
> > >>> >>>>>>> warning
> > >>> >>>>>>>>>>> stands, that I will start locking down what can get into
> > >>> >>> 1.9.0
> > >>> >>>> at
> > >>> >>>>>>> that
> > >>> >>>>>>>>>>> point.
> > >>> >>>>>>>>>>>
> > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > >>> >>>>>> bdbruin@gmail.com>
> > >>> >>>>>>>>>>> wrote:
> > >>> >>>>>>>>>>>
> > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
> > >>> >>> fixed
> > >>> >>>>>> point
> > >>> >>>>>>> in
> > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than
> to
> > a
> > >>> >>>> git
> > >>> >>>>>>> pull.
> > >>> >>>>>>>>>>>>
> > >>> >>>>>>>>>>>> Bolke
> > >>> >>>>>>>>>>>>
> > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > >>> >>>>> criccomini@apache.org
> > >>> >>>>>>>
> > >>> >>>>>>>>>>> wrote:
> > >>> >>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly
> different?
> > >>> >>>> IIRC,
> > >>> >>>>>> it's
> > >>> >>>>>>>>>>>>> basically the same, just no vote, right?
> > >>> >>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > >>> >>>>>>> bdbruin@gmail.com>
> > >>> >>>>>>>>>>>> wrote:
> > >>> >>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
> away?
> > >>> >>>>> Isn’t a
> > >>> >>>>>>>>>> beta
> > >>> >>>>>>>>>>> a
> > >>> >>>>>>>>>>>>>> bit smarter?
> > >>> >>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>> - Bolke
> > >>> >>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > >>> >>>>>> criccomini@apache.org
> > >>> >>>>>>>>
> > >>> >>>>>>>>>>>> wrote:
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> Hey all,
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
> > >>> >>> the
> > >>> >>>>>> stable
> > >>> >>>>>>>>>>> branch
> > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> > >>> >>> stable
> > >>> >>>>>> branch
> > >>> >>>>>>>>>> is
> > >>> >>>>>>>>>>>>>> cut, I
> > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked
> > into
> > >>> >>>> the
> > >>> >>>>>>>>>> branch,
> > >>> >>>>>>>>>>>> and
> > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> > >>> >>> release
> > >>> >>>>>> out.
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> Cheers,
> > >>> >>>>>>>>>>>>>>> Chris
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > >>> >>>>>>>>>>>> criccomini@apache.org
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>> wrote:
> > >>> >>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> Hey all,
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> > >>> >>> outstanding
> > >>> >>>> PRs
> > >>> >>>>>>> that
> > >>> >>>>>>>>>>> are
> > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
> Variable
> > >>> >>>>>> endpoint
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
> Airflow
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> > >>> >>> loggers
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> > >>> >>> issue
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> > >>> >>> unneeded
> > >>> >>>>> code
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
> HDFS
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> |airflow/jobs.py:create_dag_ru
> > >>> >>> n()
> > >>> >>>>>>>>>> exception
> > >>> >>>>>>>>>>>> for
> > >>> >>>>>>>>>>>>>>>> @on
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> > >>> >>> include
> > >>> >>>>>> paused
> > >>> >>>>>>>>>>> DAGs
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
> > >>> >>> not
> > >>> >>>> log
> > >>> >>>>>> to
> > >>> >>>>>>>>>>> stdout
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
> task
> > >>> >>>>>> instances
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> |airflow/jobs.py:manage_slas()
> > >>> >>>>>> exception
> > >>> >>>>>>>>>> for
> > >>> >>>>>>>>>>>>>>>> @once
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> > >>> >>> causes
> > >>> >>>> it
> > >>> >>>>>> to
> > >>> >>>>>>>>>> fail
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> > >>> >>>>>> BackfillJobTest.test_backfill_
> > >>> >>>>>>>>>>>>>> examples
> > >>> >>>>>>>>>>>>>>>> to
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> > >>> >>>>>> tests.CoreTest.test_scheduler_
> > >>> >>>>>>>>>> job
> > >>> >>>>>>>>>>>> to
> > >>> >>>>>>>>>>>>>>>> real
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> > >>> >>> matrix
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> > >>> >>> XComs
> > >>> >>>> by
> > >>> >>>>>>>>>> default
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > >>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added
> to
> > >>> >>>> hive
> > >>> >>>>>> conf
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
> into
> > >>> >>> the
> > >>> >>>>>> 1.9.0
> > >>> >>>>>>>>>>>> branch
> > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in,
> please
> > >>> >>> set
> > >>> >>>>> the
> > >>> >>>>>>> fix
> > >>> >>>>>>>>>>>>>> version
> > >>> >>>>>>>>>>>>>>>> to 1.9.0.
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> > cluster,
> > >>> >>>> and
> > >>> >>>>> it
> > >>> >>>>>>> has
> > >>> >>>>>>>>>>>> been
> > >>> >>>>>>>>>>>>>>>> running smoothly for several days.
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If
> you
> > >>> >>> run
> > >>> >>>>>>>>>> Airflow,
> > >>> >>>>>>>>>>>> it's
> > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
> branch
> > >>> >>>>>> somewhere,
> > >>> >>>>>>>>>> and
> > >>> >>>>>>>>>>>>>> verify
> > >>> >>>>>>>>>>>>>>>> it's working for your workload. **
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>>> Cheers,
> > >>> >>>>>>>>>>>>>>>> Chris
> > >>> >>>>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>>>
> > >>> >>>>>>>>>>>>
> > >>> >>>>>>>>>>>>
> > >>> >>>>>>>>>>>
> > >>> >>>>>>>>>>
> > >>> >>>>>>>>
> > >>> >>>>>>>>
> > >>> >>>>>>>
> > >>> >>>>>>
> > >>> >>>>>
> > >>> >>>>
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>>
> > >>
> > >>
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Feng Lu <fe...@google.com.INVALID>.

Hi Chris,

I know it's annoying to have last minute commit com in, but this is a
highly desirable feature for folks using GCP operators, is it possible to
include AIRFLOW-1635
<https://github.com/apache/incubator-airflow/commit/b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4>
in?
More than happy to offer help if there's something I can do.
Thanks a lot.

Feng

On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Hey all,
>
> I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
>
>   https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/
>
> The bin tarball can be installed with pip:
>
>   pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>
> The goal is to have the community install and run this to expose any bugs
> before we move on to official release candidates.
>
> Here are the outstanding blocker bugs for 1.9.0:
>
>   AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>   AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
> as
>   AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> for
> @on
>   AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
>   AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
> @once
>   AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>
> Cheers,
> Chris
>
> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >
> > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >> Works for me. Will try and cut a beta before end of week.
> >>
> >> Blockers for 1.9.0 are:
> >>
> >> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> marked
> >> as
> >> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >>
> >>
> >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >>
> >>> Hi Chris
> >>>
> >>> Can I suggest releasing a beta? The stable branch is only cut at RC
> >>> time. Betas allow us a broader exposure. It also gives us a point of
> >>> reference.
> >>>
> >>> In addition the list below are mostly longer standing issues that are
> >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
> and
> >>> 976 as blocker?
> >>>
> >>> Cheers
> >>> Bolke
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
> >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> criccomini@apache.org>
> >>> het volgende geschreven:
> >>> >
> >>> > Hey all,
> >>> >
> >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release,
> but
> >>> > seeing as there are several outstanding bugs, I'm going to delay.
> Here
> >>> are
> >>> > the bugs that I'm tracking:
> >>> >
> >>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>> marked as
> >>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> exception
> >>> for
> >>> > @on
> >>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>> stdout
> >>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> >>> for @once
> >>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email
> is
> >>> Not
> >>> > be
> >>> > AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> >>> >
> >>> > These are the priority issues. Once they're merged, I'll cut the
> >>> > v1-9-stable and beta release.
> >>> >
> >>> > If you can help clean this up, that would be really appreciated.
> >>> >
> >>> > Cheers,
> >>> > Chris
> >>> >
> >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>> criccomini@apache.org>
> >>> > wrote:
> >>> >
> >>> >> Marked it for 1.9.0.
> >>> >>
> >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi>
> >>> wrote:
> >>> >>>
> >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> >>> callbacks
> >>> >>> are not working correctly without emails... Its not a major bug,
> but
> >>> it
> >>> >>> does cause us some annoyance in our current deployment.
> >>> >>>
> >>> >>> Link to Jira:
> >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> >>> >>>
> >>> >>> Link to PR:
> >>> >>> https://github.com/apache/incubator-airflow/pull/2415
> >>> >>>
> >>> >>> Thanks!
> >>> >>> Charlie Jones
> >>> >>>
> >>> >>> CHARLIE JONES
> >>> >>> Data Engineer
> >>> >>> cjones@simpli.fi  |  M: 972.821.7631
> >>> >>> __________________________________________________
> >>> >>>
> >>> >>>
> >>> >>> Programmatic Performance.* Localized.*
> >>> >>> __________________________________________________
> >>> >>>
> >>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>> >>> 800.840.0768  |  www.simpli.fi
> >>> >>>
> >>> >>>
> >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>> criccomini@apache.org>
> >>> >>> wrote:
> >>> >>>
> >>> >>>> Merged.
> >>> >>>>
> >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>> >>> ryan.buckley@bluecore.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> Would it be possible to include AIRFLOW-1587?
> >>> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch
> >>> due
> >>> >>> to
> >>> >>>>> this issue.
> >>> >>>>>
> >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> >>> >>>>>
> >>> >>>>> Thanks,
> >>> >>>>> Ryan
> >>> >>>>>
> >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>> >>> <fokko@driesprong.frl
> >>> >>>>>
> >>> >>>>> wrote:
> >>> >>>>>
> >>> >>>>>> Hi All,
> >>> >>>>>>
> >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> >>> >>>>>>
> >>> >>>>>> Currently importing a custom logging configuration is not work
> (as
> >>> >>> far
> >>> >>>>> as I
> >>> >>>>>> know). Any feedback on the PR would also be appreciated.
> >>> >>>>>>
> >>> >>>>>> Cheers, Fokko
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> criccomini@apache.org
> >>> >:
> >>> >>>>>>
> >>> >>>>>>> Done!
> >>> >>>>>>>
> >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> >>> >>>>>>>
> >>> >>>>>>>> Can you slide the aws and emr connection type fix in?
> >>> >>>>>>>>
> >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> >>> >>>>>>>>
> >>> >>>>>>>> It keeps the connection type from getting blanked out on edit
> >>> >>> for
> >>> >>>>> these
> >>> >>>>>>>> types.
> >>> >>>>>>>>
> >>> >>>>>>>> Thanks,
> >>> >>>>>>>> Mike
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>> >>>>> criccomini@apache.org>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to
> >>> >>>> these
> >>> >>>>>>> fixes!
> >>> >>>>>>>>>
> >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>> >>>>> alex.guziel@airbnb.com
> >>> >>>>>> .
> >>> >>>>>>>> invalid
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>
> >>> >>>>>>>>>> Can we get this in?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
> >>> >>> have
> >>> >>>> in
> >>> >>>>>> 1.9.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>> >>>>>>>> criccomini@apache.org>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> >>> >>> Initial
> >>> >>>>>>> warning
> >>> >>>>>>>>>>> stands, that I will start locking down what can get into
> >>> >>> 1.9.0
> >>> >>>> at
> >>> >>>>>>> that
> >>> >>>>>>>>>>> point.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>> >>>>>> bdbruin@gmail.com>
> >>> >>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
> >>> >>> fixed
> >>> >>>>>> point
> >>> >>>>>>> in
> >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to
> a
> >>> >>>> git
> >>> >>>>>>> pull.
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>> Bolke
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>> >>>>> criccomini@apache.org
> >>> >>>>>>>
> >>> >>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly different?
> >>> >>>> IIRC,
> >>> >>>>>> it's
> >>> >>>>>>>>>>>>> basically the same, just no vote, right?
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> >>> >>>>>>> bdbruin@gmail.com>
> >>> >>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away?
> >>> >>>>> Isn’t a
> >>> >>>>>>>>>> beta
> >>> >>>>>>>>>>> a
> >>> >>>>>>>>>>>>>> bit smarter?
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>> - Bolke
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>> >>>>>> criccomini@apache.org
> >>> >>>>>>>>
> >>> >>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> Hey all,
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
> >>> >>> the
> >>> >>>>>> stable
> >>> >>>>>>>>>>> branch
> >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> >>> >>> stable
> >>> >>>>>> branch
> >>> >>>>>>>>>> is
> >>> >>>>>>>>>>>>>> cut, I
> >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked
> into
> >>> >>>> the
> >>> >>>>>>>>>> branch,
> >>> >>>>>>>>>>>> and
> >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> >>> >>> release
> >>> >>>>>> out.
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> Cheers,
> >>> >>>>>>>>>>>>>>> Chris
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> >>> >>>>>>>>>>>> criccomini@apache.org
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> Hey all,
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>> >>> outstanding
> >>> >>>> PRs
> >>> >>>>>>> that
> >>> >>>>>>>>>>> are
> >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> >>> >>>>>> endpoint
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> >>> >>> loggers
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> >>> >>> issue
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> >>> >>> unneeded
> >>> >>>>> code
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
> >>> >>> n()
> >>> >>>>>>>>>> exception
> >>> >>>>>>>>>>>> for
> >>> >>>>>>>>>>>>>>>> @on
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> >>> >>> include
> >>> >>>>>> paused
> >>> >>>>>>>>>>> DAGs
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
> >>> >>> not
> >>> >>>> log
> >>> >>>>>> to
> >>> >>>>>>>>>>> stdout
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> >>> >>>>>> instances
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> >>> >>>>>> exception
> >>> >>>>>>>>>> for
> >>> >>>>>>>>>>>>>>>> @once
> >>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> >>> >>> causes
> >>> >>>> it
> >>> >>>>>> to
> >>> >>>>>>>>>> fail
> >>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>> >>>>>> BackfillJobTest.test_backfill_
> >>> >>>>>>>>>>>>>> examples
> >>> >>>>>>>>>>>>>>>> to
> >>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>> >>>>>> tests.CoreTest.test_scheduler_
> >>> >>>>>>>>>> job
> >>> >>>>>>>>>>>> to
> >>> >>>>>>>>>>>>>>>> real
> >>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> >>> >>> matrix
> >>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> >>> >>> XComs
> >>> >>>> by
> >>> >>>>>>>>>> default
> >>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
> >>> >>>> hive
> >>> >>>>>> conf
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into
> >>> >>> the
> >>> >>>>>> 1.9.0
> >>> >>>>>>>>>>>> branch
> >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, please
> >>> >>> set
> >>> >>>>> the
> >>> >>>>>>> fix
> >>> >>>>>>>>>>>>>> version
> >>> >>>>>>>>>>>>>>>> to 1.9.0.
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
> cluster,
> >>> >>>> and
> >>> >>>>> it
> >>> >>>>>>> has
> >>> >>>>>>>>>>>> been
> >>> >>>>>>>>>>>>>>>> running smoothly for several days.
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you
> >>> >>> run
> >>> >>>>>>>>>> Airflow,
> >>> >>>>>>>>>>>> it's
> >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch
> >>> >>>>>> somewhere,
> >>> >>>>>>>>>> and
> >>> >>>>>>>>>>>>>> verify
> >>> >>>>>>>>>>>>>>>> it's working for your workload. **
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> Cheers,
> >>> >>>>>>>>>>>>>>>> Chris
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>
> >>
> >
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

I have cut a 1.9.0alpha0 release of Airflow. You can download it here:

  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/

The bin tarball can be installed with pip:

  pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz

The goal is to have the community install and run this to expose any bugs
before we move on to official release candidates.

Here are the outstanding blocker bugs for 1.9.0:

  AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
  AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
as
  AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception for
@on
  AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
  AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
@once
  AIRFLOW-976  |Bug         |Mark success running task causes it to fail

Cheers,
Chris

On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>
> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Works for me. Will try and cut a beta before end of week.
>>
>> Blockers for 1.9.0 are:
>>
>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
>> as
>> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>>
>>
>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com>
>> wrote:
>>
>>> Hi Chris
>>>
>>> Can I suggest releasing a beta? The stable branch is only cut at RC
>>> time. Betas allow us a broader exposure. It also gives us a point of
>>> reference.
>>>
>>> In addition the list below are mostly longer standing issues that are
>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and
>>> 976 as blocker?
>>>
>>> Cheers
>>> Bolke
>>>
>>> Verstuurd vanaf mijn iPad
>>>
>>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <cr...@apache.org>
>>> het volgende geschreven:
>>> >
>>> > Hey all,
>>> >
>>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
>>> > seeing as there are several outstanding bugs, I'm going to delay. Here
>>> are
>>> > the bugs that I'm tracking:
>>> >
>>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> marked as
>>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
>>> for
>>> > @on
>>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>> stdout
>>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>> for @once
>>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is
>>> Not
>>> > be
>>> > AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>>> >
>>> > These are the priority issues. Once they're merged, I'll cut the
>>> > v1-9-stable and beta release.
>>> >
>>> > If you can help clean this up, that would be really appreciated.
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>> criccomini@apache.org>
>>> > wrote:
>>> >
>>> >> Marked it for 1.9.0.
>>> >>
>>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi>
>>> wrote:
>>> >>>
>>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>>> callbacks
>>> >>> are not working correctly without emails... Its not a major bug, but
>>> it
>>> >>> does cause us some annoyance in our current deployment.
>>> >>>
>>> >>> Link to Jira:
>>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>> >>>
>>> >>> Link to PR:
>>> >>> https://github.com/apache/incubator-airflow/pull/2415
>>> >>>
>>> >>> Thanks!
>>> >>> Charlie Jones
>>> >>>
>>> >>> CHARLIE JONES
>>> >>> Data Engineer
>>> >>> cjones@simpli.fi  |  M: 972.821.7631
>>> >>> __________________________________________________
>>> >>>
>>> >>>
>>> >>> Programmatic Performance.* Localized.*
>>> >>> __________________________________________________
>>> >>>
>>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>> >>> 800.840.0768  |  www.simpli.fi
>>> >>>
>>> >>>
>>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>> criccomini@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> Merged.
>>> >>>>
>>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>> >>> ryan.buckley@bluecore.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Would it be possible to include AIRFLOW-1587?
>>> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch
>>> due
>>> >>> to
>>> >>>>> this issue.
>>> >>>>>
>>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Ryan
>>> >>>>>
>>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>> >>> <fokko@driesprong.frl
>>> >>>>>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Hi All,
>>> >>>>>>
>>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
>>> >>>>>>
>>> >>>>>> Currently importing a custom logging configuration is not work (as
>>> >>> far
>>> >>>>> as I
>>> >>>>>> know). Any feedback on the PR would also be appreciated.
>>> >>>>>>
>>> >>>>>> Cheers, Fokko
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org
>>> >:
>>> >>>>>>
>>> >>>>>>> Done!
>>> >>>>>>>
>>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>>> >>>>>>>
>>> >>>>>>>> Can you slide the aws and emr connection type fix in?
>>> >>>>>>>>
>>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>>> >>>>>>>>
>>> >>>>>>>> It keeps the connection type from getting blanked out on edit
>>> >>> for
>>> >>>>> these
>>> >>>>>>>> types.
>>> >>>>>>>>
>>> >>>>>>>> Thanks,
>>> >>>>>>>> Mike
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>> >>>>> criccomini@apache.org>
>>> >>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to
>>> >>>> these
>>> >>>>>>> fixes!
>>> >>>>>>>>>
>>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>> >>>>> alex.guziel@airbnb.com
>>> >>>>>> .
>>> >>>>>>>> invalid
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Can we get this in?
>>> >>>>>>>>>>
>>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>>> >>>>>>>>>>
>>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>> >>>>>>>>>>
>>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
>>> >>> have
>>> >>>> in
>>> >>>>>> 1.9.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>> >>>>>>>> criccomini@apache.org>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>> >>> Initial
>>> >>>>>>> warning
>>> >>>>>>>>>>> stands, that I will start locking down what can get into
>>> >>> 1.9.0
>>> >>>> at
>>> >>>>>>> that
>>> >>>>>>>>>>> point.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>> >>>>>> bdbruin@gmail.com>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
>>> >>> fixed
>>> >>>>>> point
>>> >>>>>>> in
>>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to a
>>> >>>> git
>>> >>>>>>> pull.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Bolke
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>> >>>>> criccomini@apache.org
>>> >>>>>>>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> I can do a beta. Is the process significantly different?
>>> >>>> IIRC,
>>> >>>>>> it's
>>> >>>>>>>>>>>>> basically the same, just no vote, right?
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>> >>>>>>> bdbruin@gmail.com>
>>> >>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away?
>>> >>>>> Isn’t a
>>> >>>>>>>>>> beta
>>> >>>>>>>>>>> a
>>> >>>>>>>>>>>>>> bit smarter?
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> - Bolke
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>> >>>>>> criccomini@apache.org
>>> >>>>>>>>
>>> >>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
>>> >>> the
>>> >>>>>> stable
>>> >>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
>>> >>> stable
>>> >>>>>> branch
>>> >>>>>>>>>> is
>>> >>>>>>>>>>>>>> cut, I
>>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked into
>>> >>>> the
>>> >>>>>>>>>> branch,
>>> >>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
>>> >>> release
>>> >>>>>> out.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>>> >>>>>>>>>>>> criccomini@apache.org
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>> >>> outstanding
>>> >>>> PRs
>>> >>>>>>> that
>>> >>>>>>>>>>> are
>>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
>>> >>>>>> endpoint
>>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>> >>> loggers
>>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>>> >>> issue
>>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>> >>> unneeded
>>> >>>>> code
>>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
>>> >>> n()
>>> >>>>>>>>>> exception
>>> >>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>>> @on
>>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>> >>> include
>>> >>>>>> paused
>>> >>>>>>>>>>> DAGs
>>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
>>> >>> not
>>> >>>> log
>>> >>>>>> to
>>> >>>>>>>>>>> stdout
>>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
>>> >>>>>> instances
>>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
>>> >>>>>> exception
>>> >>>>>>>>>> for
>>> >>>>>>>>>>>>>>>> @once
>>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>>> >>> causes
>>> >>>> it
>>> >>>>>> to
>>> >>>>>>>>>> fail
>>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>> >>>>>> BackfillJobTest.test_backfill_
>>> >>>>>>>>>>>>>> examples
>>> >>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>> >>>>>> tests.CoreTest.test_scheduler_
>>> >>>>>>>>>> job
>>> >>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>> real
>>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>> >>> matrix
>>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>>> >>> XComs
>>> >>>> by
>>> >>>>>>>>>> default
>>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
>>> >>>> hive
>>> >>>>>> conf
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into
>>> >>> the
>>> >>>>>> 1.9.0
>>> >>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, please
>>> >>> set
>>> >>>>> the
>>> >>>>>>> fix
>>> >>>>>>>>>>>>>> version
>>> >>>>>>>>>>>>>>>> to 1.9.0.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
>>> >>>> and
>>> >>>>> it
>>> >>>>>>> has
>>> >>>>>>>>>>>> been
>>> >>>>>>>>>>>>>>>> running smoothly for several days.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you
>>> >>> run
>>> >>>>>>>>>> Airflow,
>>> >>>>>>>>>>>> it's
>>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch
>>> >>>>>> somewhere,
>>> >>>>>>>>>> and
>>> >>>>>>>>>>>>>> verify
>>> >>>>>>>>>>>>>>>> it's working for your workload. **
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> >>
>>>
>>
>>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Welp. Work got in the way, so I'll cut the beta on Monday. :)

On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Works for me. Will try and cut a beta before end of week.
>
> Blockers for 1.9.0 are:
>
> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
> as
> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>
>
> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hi Chris
>>
>> Can I suggest releasing a beta? The stable branch is only cut at RC time.
>> Betas allow us a broader exposure. It also gives us a point of reference.
>>
>> In addition the list below are mostly longer standing issues that are
>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and
>> 976 as blocker?
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <cr...@apache.org>
>> het volgende geschreven:
>> >
>> > Hey all,
>> >
>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
>> > seeing as there are several outstanding bugs, I'm going to delay. Here
>> are
>> > the bugs that I'm tracking:
>> >
>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> marked as
>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
>> for
>> > @on
>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
>> @once
>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is
>> Not
>> > be
>> > AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>> >
>> > These are the priority issues. Once they're merged, I'll cut the
>> > v1-9-stable and beta release.
>> >
>> > If you can help clean this up, that would be really appreciated.
>> >
>> > Cheers,
>> > Chris
>> >
>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>> criccomini@apache.org>
>> > wrote:
>> >
>> >> Marked it for 1.9.0.
>> >>
>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi>
>> wrote:
>> >>>
>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>> callbacks
>> >>> are not working correctly without emails... Its not a major bug, but
>> it
>> >>> does cause us some annoyance in our current deployment.
>> >>>
>> >>> Link to Jira:
>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
>> >>>
>> >>> Link to PR:
>> >>> https://github.com/apache/incubator-airflow/pull/2415
>> >>>
>> >>> Thanks!
>> >>> Charlie Jones
>> >>>
>> >>> CHARLIE JONES
>> >>> Data Engineer
>> >>> cjones@simpli.fi  |  M: 972.821.7631
>> >>> __________________________________________________
>> >>>
>> >>>
>> >>> Programmatic Performance.* Localized.*
>> >>> __________________________________________________
>> >>>
>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> >>> 800.840.0768  |  www.simpli.fi
>> >>>
>> >>>
>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>> criccomini@apache.org>
>> >>> wrote:
>> >>>
>> >>>> Merged.
>> >>>>
>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> >>> ryan.buckley@bluecore.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Would it be possible to include AIRFLOW-1587?
>> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch due
>> >>> to
>> >>>>> this issue.
>> >>>>>
>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Ryan
>> >>>>>
>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> >>> <fokko@driesprong.frl
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi All,
>> >>>>>>
>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
>> >>>>>>
>> >>>>>> Currently importing a custom logging configuration is not work (as
>> >>> far
>> >>>>> as I
>> >>>>>> know). Any feedback on the PR would also be appreciated.
>> >>>>>>
>> >>>>>> Cheers, Fokko
>> >>>>>>
>> >>>>>>
>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org
>> >:
>> >>>>>>
>> >>>>>>> Done!
>> >>>>>>>
>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>> >>>>>>>
>> >>>>>>>> Can you slide the aws and emr connection type fix in?
>> >>>>>>>>
>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>> >>>>>>>>
>> >>>>>>>> It keeps the connection type from getting blanked out on edit
>> >>> for
>> >>>>> these
>> >>>>>>>> types.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Mike
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>> >>>>> criccomini@apache.org>
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to
>> >>>> these
>> >>>>>>> fixes!
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>> >>>>> alex.guziel@airbnb.com
>> >>>>>> .
>> >>>>>>>> invalid
>> >>>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Can we get this in?
>> >>>>>>>>>>
>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> >>>>>>>>>>
>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> >>>>>>>>>>
>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
>> >>> have
>> >>>> in
>> >>>>>> 1.9.
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>> >>>>>>>> criccomini@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>> >>> Initial
>> >>>>>>> warning
>> >>>>>>>>>>> stands, that I will start locking down what can get into
>> >>> 1.9.0
>> >>>> at
>> >>>>>>> that
>> >>>>>>>>>>> point.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>> >>>>>> bdbruin@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
>> >>> fixed
>> >>>>>> point
>> >>>>>>> in
>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to a
>> >>>> git
>> >>>>>>> pull.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Bolke
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>> >>>>> criccomini@apache.org
>> >>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I can do a beta. Is the process significantly different?
>> >>>> IIRC,
>> >>>>>> it's
>> >>>>>>>>>>>>> basically the same, just no vote, right?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>> >>>>>>> bdbruin@gmail.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away?
>> >>>>> Isn’t a
>> >>>>>>>>>> beta
>> >>>>>>>>>>> a
>> >>>>>>>>>>>>>> bit smarter?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> - Bolke
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>> >>>>>> criccomini@apache.org
>> >>>>>>>>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
>> >>> the
>> >>>>>> stable
>> >>>>>>>>>>> branch
>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
>> >>> stable
>> >>>>>> branch
>> >>>>>>>>>> is
>> >>>>>>>>>>>>>> cut, I
>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked into
>> >>>> the
>> >>>>>>>>>> branch,
>> >>>>>>>>>>>> and
>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
>> >>> release
>> >>>>>> out.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>> >>>>>>>>>>>> criccomini@apache.org
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>> >>> outstanding
>> >>>> PRs
>> >>>>>>> that
>> >>>>>>>>>>> are
>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
>> >>>>>> endpoint
>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>> >>> loggers
>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>> >>> issue
>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>> >>> unneeded
>> >>>>> code
>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
>> >>> n()
>> >>>>>>>>>> exception
>> >>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>> @on
>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>> >>> include
>> >>>>>> paused
>> >>>>>>>>>>> DAGs
>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
>> >>> not
>> >>>> log
>> >>>>>> to
>> >>>>>>>>>>> stdout
>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
>> >>>>>> instances
>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
>> >>>>>> exception
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>>>> @once
>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>> >>> causes
>> >>>> it
>> >>>>>> to
>> >>>>>>>>>> fail
>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>> >>>>>> BackfillJobTest.test_backfill_
>> >>>>>>>>>>>>>> examples
>> >>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>> >>>>>> tests.CoreTest.test_scheduler_
>> >>>>>>>>>> job
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> real
>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>> >>> matrix
>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>> >>> XComs
>> >>>> by
>> >>>>>>>>>> default
>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
>> >>>> hive
>> >>>>>> conf
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into
>> >>> the
>> >>>>>> 1.9.0
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, please
>> >>> set
>> >>>>> the
>> >>>>>>> fix
>> >>>>>>>>>>>>>> version
>> >>>>>>>>>>>>>>>> to 1.9.0.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
>> >>>> and
>> >>>>> it
>> >>>>>>> has
>> >>>>>>>>>>>> been
>> >>>>>>>>>>>>>>>> running smoothly for several days.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you
>> >>> run
>> >>>>>>>>>> Airflow,
>> >>>>>>>>>>>> it's
>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch
>> >>>>>> somewhere,
>> >>>>>>>>>> and
>> >>>>>>>>>>>>>> verify
>> >>>>>>>>>>>>>>>> it's working for your workload. **
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>>
>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Works for me. Will try and cut a beta before end of week.

Blockers for 1.9.0 are:

AIRFLOW-1611 |Bug         |Customize logging in Airflow
AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
AIRFLOW-976  |Bug         |Mark success running task causes it to fail


On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bd...@gmail.com> wrote:

> Hi Chris
>
> Can I suggest releasing a beta? The stable branch is only cut at RC time.
> Betas allow us a broader exposure. It also gives us a point of reference.
>
> In addition the list below are mostly longer standing issues that are also
> part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and 976 as
> blocker?
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <cr...@apache.org>
> het volgende geschreven:
> >
> > Hey all,
> >
> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
> > seeing as there are several outstanding bugs, I'm going to delay. Here
> are
> > the bugs that I'm tracking:
> >
> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> marked as
> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> for
> > @on
> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
> @once
> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is
> Not
> > be
> > AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >
> > These are the priority issues. Once they're merged, I'll cut the
> > v1-9-stable and beta release.
> >
> > If you can help clean this up, that would be really appreciated.
> >
> > Cheers,
> > Chris
> >
> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> Marked it for 1.9.0.
> >>
> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi>
> wrote:
> >>>
> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> callbacks
> >>> are not working correctly without emails... Its not a major bug, but it
> >>> does cause us some annoyance in our current deployment.
> >>>
> >>> Link to Jira:
> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> >>>
> >>> Link to PR:
> >>> https://github.com/apache/incubator-airflow/pull/2415
> >>>
> >>> Thanks!
> >>> Charlie Jones
> >>>
> >>> CHARLIE JONES
> >>> Data Engineer
> >>> cjones@simpli.fi  |  M: 972.821.7631
> >>> __________________________________________________
> >>>
> >>>
> >>> Programmatic Performance.* Localized.*
> >>> __________________________________________________
> >>>
> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>> 800.840.0768  |  www.simpli.fi
> >>>
> >>>
> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> criccomini@apache.org>
> >>> wrote:
> >>>
> >>>> Merged.
> >>>>
> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>> ryan.buckley@bluecore.com>
> >>>> wrote:
> >>>>
> >>>>> Would it be possible to include AIRFLOW-1587?
> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch due
> >>> to
> >>>>> this issue.
> >>>>>
> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> >>>>>
> >>>>> Thanks,
> >>>>> Ryan
> >>>>>
> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>> <fokko@driesprong.frl
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> >>>>>>
> >>>>>> Currently importing a custom logging configuration is not work (as
> >>> far
> >>>>> as I
> >>>>>> know). Any feedback on the PR would also be appreciated.
> >>>>>>
> >>>>>> Cheers, Fokko
> >>>>>>
> >>>>>>
> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
> >>>>>>
> >>>>>>> Done!
> >>>>>>>
> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> >>>>>>>
> >>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> >>>>>>>>
> >>>>>>>> It keeps the connection type from getting blanked out on edit
> >>> for
> >>>>> these
> >>>>>>>> types.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Mike
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>> criccomini@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to
> >>>> these
> >>>>>>> fixes!
> >>>>>>>>>
> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>> alex.guziel@airbnb.com
> >>>>>> .
> >>>>>>>> invalid
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Can we get this in?
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>>>>>>>>>
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>
> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
> >>> have
> >>>> in
> >>>>>> 1.9.
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> >>>>>>>> criccomini@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
> >>> Initial
> >>>>>>> warning
> >>>>>>>>>>> stands, that I will start locking down what can get into
> >>> 1.9.0
> >>>> at
> >>>>>>> that
> >>>>>>>>>>> point.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> >>>>>> bdbruin@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
> >>> fixed
> >>>>>> point
> >>>>>>> in
> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to a
> >>>> git
> >>>>>>> pull.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Bolke
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>> criccomini@apache.org
> >>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I can do a beta. Is the process significantly different?
> >>>> IIRC,
> >>>>>> it's
> >>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away?
> >>>>> Isn’t a
> >>>>>>>>>> beta
> >>>>>>>>>>> a
> >>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>> criccomini@apache.org
> >>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
> >>> the
> >>>>>> stable
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
> >>> stable
> >>>>>> branch
> >>>>>>>>>> is
> >>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked into
> >>>> the
> >>>>>>>>>> branch,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the
> >>> release
> >>>>>> out.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
> >>> outstanding
> >>>> PRs
> >>>>>>> that
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> >>>>>> endpoint
> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
> >>> loggers
> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> >>> issue
> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> >>> unneeded
> >>>>> code
> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
> >>> n()
> >>>>>>>>>> exception
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> >>> include
> >>>>>> paused
> >>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
> >>> not
> >>>> log
> >>>>>> to
> >>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> >>>>>> instances
> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> >>>>>> exception
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> >>> causes
> >>>> it
> >>>>>> to
> >>>>>>>>>> fail
> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>> job
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
> >>> matrix
> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
> >>> XComs
> >>>> by
> >>>>>>>>>> default
> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
> >>>> hive
> >>>>>> conf
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into
> >>> the
> >>>>>> 1.9.0
> >>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, please
> >>> set
> >>>>> the
> >>>>>>> fix
> >>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
> >>>> and
> >>>>> it
> >>>>>>> has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you
> >>> run
> >>>>>>>>>> Airflow,
> >>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch
> >>>>>> somewhere,
> >>>>>>>>>> and
> >>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

Hi Chris

Can I suggest releasing a beta? The stable branch is only cut at RC time. Betas allow us a broader exposure. It also gives us a point of reference.

In addition the list below are mostly longer standing issues that are also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and 976 as blocker?

Cheers
Bolke

Verstuurd vanaf mijn iPad

> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <cr...@apache.org> het volgende geschreven:
> 
> Hey all,
> 
> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
> seeing as there are several outstanding bugs, I'm going to delay. Here are
> the bugs that I'm tracking:
> 
> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception for
> @on
> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for @once
> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is Not
> be
> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> 
> These are the priority issues. Once they're merged, I'll cut the
> v1-9-stable and beta release.
> 
> If you can help clean this up, that would be really appreciated.
> 
> Cheers,
> Chris
> 
> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <cr...@apache.org>
> wrote:
> 
>> Marked it for 1.9.0.
>> 
>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi> wrote:
>>> 
>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
>>> are not working correctly without emails... Its not a major bug, but it
>>> does cause us some annoyance in our current deployment.
>>> 
>>> Link to Jira:
>>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>> 
>>> Link to PR:
>>> https://github.com/apache/incubator-airflow/pull/2415
>>> 
>>> Thanks!
>>> Charlie Jones
>>> 
>>> CHARLIE JONES
>>> Data Engineer
>>> cjones@simpli.fi  |  M: 972.821.7631
>>> __________________________________________________
>>> 
>>> 
>>> Programmatic Performance.* Localized.*
>>> __________________________________________________
>>> 
>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>> 800.840.0768  |  www.simpli.fi
>>> 
>>> 
>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>> 
>>>> Merged.
>>>> 
>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>> ryan.buckley@bluecore.com>
>>>> wrote:
>>>> 
>>>>> Would it be possible to include AIRFLOW-1587?
>>>>> Running dags from the UI is currently broken on the 1.9.0 branch due
>>> to
>>>>> this issue.
>>>>> 
>>>>> https://github.com/apache/incubator-airflow/pull/2590
>>>>> 
>>>>> Thanks,
>>>>> Ryan
>>>>> 
>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>> <fokko@driesprong.frl
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>>>>> https://github.com/apache/incubator-airflow/pull/2631
>>>>>> 
>>>>>> Currently importing a custom logging configuration is not work (as
>>> far
>>>>> as I
>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>> 
>>>>>> Cheers, Fokko
>>>>>> 
>>>>>> 
>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
>>>>>> 
>>>>>>> Done!
>>>>>>> 
>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>>>>>>> 
>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>> 
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>>>>>>>> 
>>>>>>>> It keeps the connection type from getting blanked out on edit
>>> for
>>>>> these
>>>>>>>> types.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to
>>>> these
>>>>>>> fixes!
>>>>>>>>> 
>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>> alex.guziel@airbnb.com
>>>>>> .
>>>>>>>> invalid
>>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Can we get this in?
>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>> 
>>>>>>>>>> It speeds up loading times a lot, so it's a good thing to
>>> have
>>>> in
>>>>>> 1.9.
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>> Initial
>>>>>>> warning
>>>>>>>>>>> stands, that I will start locking down what can get into
>>> 1.9.0
>>>> at
>>>>>>> that
>>>>>>>>>>> point.
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>> bdbruin@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> No vote indeed, just to gather feedback on a particular
>>> fixed
>>>>>> point
>>>>>>> in
>>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to a
>>>> git
>>>>>>> pull.
>>>>>>>>>>>> 
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>> criccomini@apache.org
>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I can do a beta. Is the process significantly different?
>>>> IIRC,
>>>>>> it's
>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away?
>>>>> Isn’t a
>>>>>>>>>> beta
>>>>>>>>>>> a
>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>> criccomini@apache.org
>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut
>>> the
>>>>>> stable
>>>>>>>>>>> branch
>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the
>>> stable
>>>>>> branch
>>>>>>>>>> is
>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>> will be locking down what commits get cherry picked into
>>>> the
>>>>>>>>>> branch,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> will only be doing PRs that are required to get the
>>> release
>>>>>> out.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>> outstanding
>>>> PRs
>>>>>>> that
>>>>>>>>>>> are
>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
>>>>>> endpoint
>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>> loggers
>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>>> issue
>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>> unneeded
>>>>> code
>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
>>> n()
>>>>>>>>>> exception
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>> include
>>>>>> paused
>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
>>> not
>>>> log
>>>>>> to
>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
>>>>>> instances
>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
>>>>>> exception
>>>>>>>>>> for
>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>>> causes
>>>> it
>>>>>> to
>>>>>>>>>> fail
>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>> job
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>> matrix
>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>>> XComs
>>>> by
>>>>>>>>>> default
>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
>>>> hive
>>>>>> conf
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into
>>> the
>>>>>> 1.9.0
>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>> shortly, so if you have something you want in, please
>>> set
>>>>> the
>>>>>>> fix
>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
>>>> and
>>>>> it
>>>>>>> has
>>>>>>>>>>>> been
>>>>>>>>>>>>>>>> running smoothly for several days.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you
>>> run
>>>>>>>>>> Airflow,
>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch
>>>>>> somewhere,
>>>>>>>>>> and
>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>> it's working for your workload. **
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
seeing as there are several outstanding bugs, I'm going to delay. Here are
the bugs that I'm tracking:

AIRFLOW-1611 |Bug         |Customize logging in Airflow
AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception for
@on
AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for @once
AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is Not
be
AIRFLOW-976  |Bug         |Mark success running task causes it to fail

These are the priority issues. Once they're merged, I'll cut the
v1-9-stable and beta release.

If you can help clean this up, that would be really appreciated.

Cheers,
Chris

On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Marked it for 1.9.0.
>
> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi> wrote:
>
>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
>> are not working correctly without emails... Its not a major bug, but it
>> does cause us some annoyance in our current deployment.
>>
>> Link to Jira:
>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>
>> Link to PR:
>> https://github.com/apache/incubator-airflow/pull/2415
>>
>> Thanks!
>> Charlie Jones
>>
>> CHARLIE JONES
>> Data Engineer
>> cjones@simpli.fi  |  M: 972.821.7631
>> __________________________________________________
>>
>>
>> Programmatic Performance.* Localized.*
>> __________________________________________________
>>
>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> 800.840.0768  |  www.simpli.fi
>>
>>
>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>> > Merged.
>> >
>> > On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> ryan.buckley@bluecore.com>
>> > wrote:
>> >
>> > > Would it be possible to include AIRFLOW-1587?
>> > > Running dags from the UI is currently broken on the 1.9.0 branch due
>> to
>> > > this issue.
>> > >
>> > > https://github.com/apache/incubator-airflow/pull/2590
>> > >
>> > > Thanks,
>> > > Ryan
>> > >
>> > > On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> <fokko@driesprong.frl
>> > >
>> > > wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > I would like to include AIRFLOW-1611 in the 1.9.0 release:
>> > > > https://github.com/apache/incubator-airflow/pull/2631
>> > > >
>> > > > Currently importing a custom logging configuration is not work (as
>> far
>> > > as I
>> > > > know). Any feedback on the PR would also be appreciated.
>> > > >
>> > > > Cheers, Fokko
>> > > >
>> > > >
>> > > > 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
>> > > >
>> > > > > Done!
>> > > > >
>> > > > > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> > > > > michael.crawford@modernizingmedicine.com> wrote:
>> > > > >
>> > > > > > Can you slide the aws and emr connection type fix in?
>> > > > > >
>> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
>> > > > > > https://github.com/apache/incubator-airflow/pull/2626 <
>> > > > > > https://github.com/apache/incubator-airflow/pull/2626>
>> > > > > >
>> > > > > > It keeps the connection type from getting blanked out on edit
>> for
>> > > these
>> > > > > > types.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mike
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>> > > criccomini@apache.org>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Absolutely. Just cherry-picked. I've been looking forward to
>> > these
>> > > > > fixes!
>> > > > > > >
>> > > > > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>> > > alex.guziel@airbnb.com
>> > > > .
>> > > > > > invalid
>> > > > > > >> wrote:
>> > > > > > >
>> > > > > > >> Can we get this in?
>> > > > > > >>
>> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> > > > > > >>
>> > > > > > >> https://github.com/apache/incubator-airflow/commit/
>> > > > > > >> b6d2e0a46978e93e16576604624f57d1388814f2
>> > > > > > >> https://github.com/apache/incubator-airflow/commit/
>> > > > > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> > > > > > >>
>> > > > > > >> It speeds up loading times a lot, so it's a good thing to
>> have
>> > in
>> > > > 1.9.
>> > > > > > >>
>> > > > > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>> > > > > > criccomini@apache.org>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> Sounds good. I'll plan on stable+beta next week, then.
>> Initial
>> > > > > warning
>> > > > > > >>> stands, that I will start locking down what can get into
>> 1.9.0
>> > at
>> > > > > that
>> > > > > > >>> point.
>> > > > > > >>>
>> > > > > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>> > > > bdbruin@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>>
>> > > > > > >>>> No vote indeed, just to gather feedback on a particular
>> fixed
>> > > > point
>> > > > > in
>> > > > > > >>>> time. It also gives a bit more trust to a tarball than to a
>> > git
>> > > > > pull.
>> > > > > > >>>>
>> > > > > > >>>> Bolke
>> > > > > > >>>>
>> > > > > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>> > > criccomini@apache.org
>> > > > >
>> > > > > > >>> wrote:
>> > > > > > >>>>>
>> > > > > > >>>>> I can do a beta. Is the process significantly different?
>> > IIRC,
>> > > > it's
>> > > > > > >>>>> basically the same, just no vote, right?
>> > > > > > >>>>>
>> > > > > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>> > > > > bdbruin@gmail.com>
>> > > > > > >>>> wrote:
>> > > > > > >>>>>
>> > > > > > >>>>>> Are you sure you want to go ahead and do RCs right away?
>> > > Isn’t a
>> > > > > > >> beta
>> > > > > > >>> a
>> > > > > > >>>>>> bit smarter?
>> > > > > > >>>>>>
>> > > > > > >>>>>> - Bolke
>> > > > > > >>>>>>
>> > > > > > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>> > > > criccomini@apache.org
>> > > > > >
>> > > > > > >>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Hey all,
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> I want to send out a warning that I'm planning to cut
>> the
>> > > > stable
>> > > > > > >>> branch
>> > > > > > >>>>>>> next week, and begin the RC1 release vote. Once the
>> stable
>> > > > branch
>> > > > > > >> is
>> > > > > > >>>>>> cut, I
>> > > > > > >>>>>>> will be locking down what commits get cherry picked into
>> > the
>> > > > > > >> branch,
>> > > > > > >>>> and
>> > > > > > >>>>>>> will only be doing PRs that are required to get the
>> release
>> > > > out.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Cheers,
>> > > > > > >>>>>>> Chris
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>> > > > > > >>>> criccomini@apache.org
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> Hey all,
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> An update on the 1.9.0 release. Here are the
>> outstanding
>> > PRs
>> > > > > that
>> > > > > > >>> are
>> > > > > > >>>>>>>> slated to be included into 1.9.0:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>> > > > > > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
>> > > > endpoint
>> > > > > > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>> > > > > > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>> loggers
>> > > > > > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>> > > > > > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
>> issue
>> > > > > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>> unneeded
>> > > code
>> > > > > > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>> > > > > > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
>> n()
>> > > > > > >> exception
>> > > > > > >>>> for
>> > > > > > >>>>>>>> @on
>> > > > > > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>> include
>> > > > paused
>> > > > > > >>> DAGs
>> > > > > > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can
>> not
>> > log
>> > > > to
>> > > > > > >>> stdout
>> > > > > > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
>> > > > instances
>> > > > > > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
>> > > > exception
>> > > > > > >> for
>> > > > > > >>>>>>>> @once
>> > > > > > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task
>> causes
>> > it
>> > > > to
>> > > > > > >> fail
>> > > > > > >>>>>>>> AIRFLOW-914  |Open      |Refactor
>> > > > BackfillJobTest.test_backfill_
>> > > > > > >>>>>> examples
>> > > > > > >>>>>>>> to
>> > > > > > >>>>>>>> AIRFLOW-913  |Open      |Refactor
>> > > > tests.CoreTest.test_scheduler_
>> > > > > > >> job
>> > > > > > >>>> to
>> > > > > > >>>>>>>> real
>> > > > > > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>> matrix
>> > > > > > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push
>> XComs
>> > by
>> > > > > > >> default
>> > > > > > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>> > > > > > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>> > > > > > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
>> > hive
>> > > > conf
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> I will be locking down what can get cherry-picked into
>> the
>> > > > 1.9.0
>> > > > > > >>>> branch
>> > > > > > >>>>>>>> shortly, so if you have something you want in, please
>> set
>> > > the
>> > > > > fix
>> > > > > > >>>>>> version
>> > > > > > >>>>>>>> to 1.9.0.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
>> > and
>> > > it
>> > > > > has
>> > > > > > >>>> been
>> > > > > > >>>>>>>> running smoothly for several days.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> ** I could really use help verifying stability. If you
>> run
>> > > > > > >> Airflow,
>> > > > > > >>>> it's
>> > > > > > >>>>>>>> in your best interest to deploy the 1.9.0 test branch
>> > > > somewhere,
>> > > > > > >> and
>> > > > > > >>>>>> verify
>> > > > > > >>>>>>>> it's working for your workload. **
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Cheers,
>> > > > > > >>>>>>>> Chris
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>
>> > > > > > >>>>
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Marked it for 1.9.0.

On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cj...@simpli.fi> wrote:

> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
> are not working correctly without emails... Its not a major bug, but it
> does cause us some annoyance in our current deployment.
>
> Link to Jira:
> https://issues.apache.org/jira/browse/AIRFLOW-988
>
> Link to PR:
> https://github.com/apache/incubator-airflow/pull/2415
>
> Thanks!
> Charlie Jones
>
> CHARLIE JONES
> Data Engineer
> cjones@simpli.fi  |  M: 972.821.7631
> __________________________________________________
>
>
> Programmatic Performance.* Localized.*
> __________________________________________________
>
> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> 800.840.0768  |  www.simpli.fi
>
>
> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Merged.
> >
> > On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <ryan.buckley@bluecore.com
> >
> > wrote:
> >
> > > Would it be possible to include AIRFLOW-1587?
> > > Running dags from the UI is currently broken on the 1.9.0 branch due to
> > > this issue.
> > >
> > > https://github.com/apache/incubator-airflow/pull/2590
> > >
> > > Thanks,
> > > Ryan
> > >
> > > On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > > > https://github.com/apache/incubator-airflow/pull/2631
> > > >
> > > > Currently importing a custom logging configuration is not work (as
> far
> > > as I
> > > > know). Any feedback on the PR would also be appreciated.
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > > 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
> > > >
> > > > > Done!
> > > > >
> > > > > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > > > > michael.crawford@modernizingmedicine.com> wrote:
> > > > >
> > > > > > Can you slide the aws and emr connection type fix in?
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > > > > > https://github.com/apache/incubator-airflow/pull/2626 <
> > > > > > https://github.com/apache/incubator-airflow/pull/2626>
> > > > > >
> > > > > > It keeps the connection type from getting blanked out on edit for
> > > these
> > > > > > types.
> > > > > >
> > > > > > Thanks,
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > > criccomini@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Absolutely. Just cherry-picked. I've been looking forward to
> > these
> > > > > fixes!
> > > > > > >
> > > > > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > > alex.guziel@airbnb.com
> > > > .
> > > > > > invalid
> > > > > > >> wrote:
> > > > > > >
> > > > > > >> Can we get this in?
> > > > > > >>
> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > > > > > >>
> > > > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > > > >> b6d2e0a46978e93e16576604624f57d1388814f2
> > > > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > > > > > >>
> > > > > > >> It speeds up loading times a lot, so it's a good thing to have
> > in
> > > > 1.9.
> > > > > > >>
> > > > > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > > > > > criccomini@apache.org>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Sounds good. I'll plan on stable+beta next week, then.
> Initial
> > > > > warning
> > > > > > >>> stands, that I will start locking down what can get into
> 1.9.0
> > at
> > > > > that
> > > > > > >>> point.
> > > > > > >>>
> > > > > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > > > bdbruin@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> No vote indeed, just to gather feedback on a particular
> fixed
> > > > point
> > > > > in
> > > > > > >>>> time. It also gives a bit more trust to a tarball than to a
> > git
> > > > > pull.
> > > > > > >>>>
> > > > > > >>>> Bolke
> > > > > > >>>>
> > > > > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > > criccomini@apache.org
> > > > >
> > > > > > >>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> I can do a beta. Is the process significantly different?
> > IIRC,
> > > > it's
> > > > > > >>>>> basically the same, just no vote, right?
> > > > > > >>>>>
> > > > > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > > > > bdbruin@gmail.com>
> > > > > > >>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Are you sure you want to go ahead and do RCs right away?
> > > Isn’t a
> > > > > > >> beta
> > > > > > >>> a
> > > > > > >>>>>> bit smarter?
> > > > > > >>>>>>
> > > > > > >>>>>> - Bolke
> > > > > > >>>>>>
> > > > > > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > > > criccomini@apache.org
> > > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>> Hey all,
> > > > > > >>>>>>>
> > > > > > >>>>>>> I want to send out a warning that I'm planning to cut the
> > > > stable
> > > > > > >>> branch
> > > > > > >>>>>>> next week, and begin the RC1 release vote. Once the
> stable
> > > > branch
> > > > > > >> is
> > > > > > >>>>>> cut, I
> > > > > > >>>>>>> will be locking down what commits get cherry picked into
> > the
> > > > > > >> branch,
> > > > > > >>>> and
> > > > > > >>>>>>> will only be doing PRs that are required to get the
> release
> > > > out.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Cheers,
> > > > > > >>>>>>> Chris
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > > > > >>>> criccomini@apache.org
> > > > > > >>>>>>>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Hey all,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> An update on the 1.9.0 release. Here are the outstanding
> > PRs
> > > > > that
> > > > > > >>> are
> > > > > > >>>>>>>> slated to be included into 1.9.0:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > > > > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> > > > endpoint
> > > > > > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > > > > > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > > > > > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > > > > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE
> issue
> > > > > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
> unneeded
> > > code
> > > > > > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > > > > > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_
> run()
> > > > > > >> exception
> > > > > > >>>> for
> > > > > > >>>>>>>> @on
> > > > > > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
> include
> > > > paused
> > > > > > >>> DAGs
> > > > > > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not
> > log
> > > > to
> > > > > > >>> stdout
> > > > > > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> > > > instances
> > > > > > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> > > > exception
> > > > > > >> for
> > > > > > >>>>>>>> @once
> > > > > > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task
> causes
> > it
> > > > to
> > > > > > >> fail
> > > > > > >>>>>>>> AIRFLOW-914  |Open      |Refactor
> > > > BackfillJobTest.test_backfill_
> > > > > > >>>>>> examples
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>> AIRFLOW-913  |Open      |Refactor
> > > > tests.CoreTest.test_scheduler_
> > > > > > >> job
> > > > > > >>>> to
> > > > > > >>>>>>>> real
> > > > > > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > > > > > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs
> > by
> > > > > > >> default
> > > > > > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > > > > > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > > > > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
> > hive
> > > > conf
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I will be locking down what can get cherry-picked into
> the
> > > > 1.9.0
> > > > > > >>>> branch
> > > > > > >>>>>>>> shortly, so if you have something you want in, please
> set
> > > the
> > > > > fix
> > > > > > >>>>>> version
> > > > > > >>>>>>>> to 1.9.0.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
> > and
> > > it
> > > > > has
> > > > > > >>>> been
> > > > > > >>>>>>>> running smoothly for several days.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> ** I could really use help verifying stability. If you
> run
> > > > > > >> Airflow,
> > > > > > >>>> it's
> > > > > > >>>>>>>> in your best interest to deploy the 1.9.0 test branch
> > > > somewhere,
> > > > > > >> and
> > > > > > >>>>>> verify
> > > > > > >>>>>>>> it's working for your workload. **
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Cheers,
> > > > > > >>>>>>>> Chris
> > > > > > >>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Charlie Jones <cj...@simpli.fi>.

Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
are not working correctly without emails... Its not a major bug, but it
does cause us some annoyance in our current deployment.

Link to Jira:
https://issues.apache.org/jira/browse/AIRFLOW-988

Link to PR:
https://github.com/apache/incubator-airflow/pull/2415

Thanks!
Charlie Jones

CHARLIE JONES
Data Engineer
cjones@simpli.fi  |  M: 972.821.7631
__________________________________________________


Programmatic Performance.* Localized.*
__________________________________________________

1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
800.840.0768  |  www.simpli.fi


On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Merged.
>
> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <ry...@bluecore.com>
> wrote:
>
> > Would it be possible to include AIRFLOW-1587?
> > Running dags from the UI is currently broken on the 1.9.0 branch due to
> > this issue.
> >
> > https://github.com/apache/incubator-airflow/pull/2590
> >
> > Thanks,
> > Ryan
> >
> > On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko <fokko@driesprong.frl
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > > https://github.com/apache/incubator-airflow/pull/2631
> > >
> > > Currently importing a custom logging configuration is not work (as far
> > as I
> > > know). Any feedback on the PR would also be appreciated.
> > >
> > > Cheers, Fokko
> > >
> > >
> > > 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
> > >
> > > > Done!
> > > >
> > > > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > > > michael.crawford@modernizingmedicine.com> wrote:
> > > >
> > > > > Can you slide the aws and emr connection type fix in?
> > > > >
> > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > > > > https://github.com/apache/incubator-airflow/pull/2626 <
> > > > > https://github.com/apache/incubator-airflow/pull/2626>
> > > > >
> > > > > It keeps the connection type from getting blanked out on edit for
> > these
> > > > > types.
> > > > >
> > > > > Thanks,
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> > criccomini@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > Absolutely. Just cherry-picked. I've been looking forward to
> these
> > > > fixes!
> > > > > >
> > > > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> > alex.guziel@airbnb.com
> > > .
> > > > > invalid
> > > > > >> wrote:
> > > > > >
> > > > > >> Can we get this in?
> > > > > >>
> > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > > > > >>
> > > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > > >> b6d2e0a46978e93e16576604624f57d1388814f2
> > > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > > > > >>
> > > > > >> It speeds up loading times a lot, so it's a good thing to have
> in
> > > 1.9.
> > > > > >>
> > > > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > > > > criccomini@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Sounds good. I'll plan on stable+beta next week, then. Initial
> > > > warning
> > > > > >>> stands, that I will start locking down what can get into 1.9.0
> at
> > > > that
> > > > > >>> point.
> > > > > >>>
> > > > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > > bdbruin@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> No vote indeed, just to gather feedback on a particular fixed
> > > point
> > > > in
> > > > > >>>> time. It also gives a bit more trust to a tarball than to a
> git
> > > > pull.
> > > > > >>>>
> > > > > >>>> Bolke
> > > > > >>>>
> > > > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> > criccomini@apache.org
> > > >
> > > > > >>> wrote:
> > > > > >>>>>
> > > > > >>>>> I can do a beta. Is the process significantly different?
> IIRC,
> > > it's
> > > > > >>>>> basically the same, just no vote, right?
> > > > > >>>>>
> > > > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > > > bdbruin@gmail.com>
> > > > > >>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Are you sure you want to go ahead and do RCs right away?
> > Isn’t a
> > > > > >> beta
> > > > > >>> a
> > > > > >>>>>> bit smarter?
> > > > > >>>>>>
> > > > > >>>>>> - Bolke
> > > > > >>>>>>
> > > > > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > > criccomini@apache.org
> > > > >
> > > > > >>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> Hey all,
> > > > > >>>>>>>
> > > > > >>>>>>> I want to send out a warning that I'm planning to cut the
> > > stable
> > > > > >>> branch
> > > > > >>>>>>> next week, and begin the RC1 release vote. Once the stable
> > > branch
> > > > > >> is
> > > > > >>>>>> cut, I
> > > > > >>>>>>> will be locking down what commits get cherry picked into
> the
> > > > > >> branch,
> > > > > >>>> and
> > > > > >>>>>>> will only be doing PRs that are required to get the release
> > > out.
> > > > > >>>>>>>
> > > > > >>>>>>> Cheers,
> > > > > >>>>>>> Chris
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > > > >>>> criccomini@apache.org
> > > > > >>>>>>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Hey all,
> > > > > >>>>>>>>
> > > > > >>>>>>>> An update on the 1.9.0 release. Here are the outstanding
> PRs
> > > > that
> > > > > >>> are
> > > > > >>>>>>>> slated to be included into 1.9.0:
> > > > > >>>>>>>>
> > > > > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > > > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> > > endpoint
> > > > > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > > > > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > > > > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > > > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > > > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded
> > code
> > > > > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > > > > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> > > > > >> exception
> > > > > >>>> for
> > > > > >>>>>>>> @on
> > > > > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include
> > > paused
> > > > > >>> DAGs
> > > > > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not
> log
> > > to
> > > > > >>> stdout
> > > > > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> > > instances
> > > > > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> > > exception
> > > > > >> for
> > > > > >>>>>>>> @once
> > > > > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes
> it
> > > to
> > > > > >> fail
> > > > > >>>>>>>> AIRFLOW-914  |Open      |Refactor
> > > BackfillJobTest.test_backfill_
> > > > > >>>>>> examples
> > > > > >>>>>>>> to
> > > > > >>>>>>>> AIRFLOW-913  |Open      |Refactor
> > > tests.CoreTest.test_scheduler_
> > > > > >> job
> > > > > >>>> to
> > > > > >>>>>>>> real
> > > > > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > > > > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs
> by
> > > > > >> default
> > > > > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > > > > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > > > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to
> hive
> > > conf
> > > > > >>>>>>>>
> > > > > >>>>>>>> I will be locking down what can get cherry-picked into the
> > > 1.9.0
> > > > > >>>> branch
> > > > > >>>>>>>> shortly, so if you have something you want in, please set
> > the
> > > > fix
> > > > > >>>>>> version
> > > > > >>>>>>>> to 1.9.0.
> > > > > >>>>>>>>
> > > > > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster,
> and
> > it
> > > > has
> > > > > >>>> been
> > > > > >>>>>>>> running smoothly for several days.
> > > > > >>>>>>>>
> > > > > >>>>>>>> ** I could really use help verifying stability. If you run
> > > > > >> Airflow,
> > > > > >>>> it's
> > > > > >>>>>>>> in your best interest to deploy the 1.9.0 test branch
> > > somewhere,
> > > > > >> and
> > > > > >>>>>> verify
> > > > > >>>>>>>> it's working for your workload. **
> > > > > >>>>>>>>
> > > > > >>>>>>>> Cheers,
> > > > > >>>>>>>> Chris
> > > > > >>>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Merged.

On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <ry...@bluecore.com>
wrote:

> Would it be possible to include AIRFLOW-1587?
> Running dags from the UI is currently broken on the 1.9.0 branch due to
> this issue.
>
> https://github.com/apache/incubator-airflow/pull/2590
>
> Thanks,
> Ryan
>
> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi All,
> >
> > I would like to include AIRFLOW-1611 in the 1.9.0 release:
> > https://github.com/apache/incubator-airflow/pull/2631
> >
> > Currently importing a custom logging configuration is not work (as far
> as I
> > know). Any feedback on the PR would also be appreciated.
> >
> > Cheers, Fokko
> >
> >
> > 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
> >
> > > Done!
> > >
> > > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > > michael.crawford@modernizingmedicine.com> wrote:
> > >
> > > > Can you slide the aws and emr connection type fix in?
> > > >
> > > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > > > https://github.com/apache/incubator-airflow/pull/2626 <
> > > > https://github.com/apache/incubator-airflow/pull/2626>
> > > >
> > > > It keeps the connection type from getting blanked out on edit for
> these
> > > > types.
> > > >
> > > > Thanks,
> > > > Mike
> > > >
> > > >
> > > >
> > > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> criccomini@apache.org>
> > > > wrote:
> > > > >
> > > > > Absolutely. Just cherry-picked. I've been looking forward to these
> > > fixes!
> > > > >
> > > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> alex.guziel@airbnb.com
> > .
> > > > invalid
> > > > >> wrote:
> > > > >
> > > > >> Can we get this in?
> > > > >>
> > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > > > >>
> > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > >> b6d2e0a46978e93e16576604624f57d1388814f2
> > > > >> https://github.com/apache/incubator-airflow/commit/
> > > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > > > >>
> > > > >> It speeds up loading times a lot, so it's a good thing to have in
> > 1.9.
> > > > >>
> > > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > > > criccomini@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >>> Sounds good. I'll plan on stable+beta next week, then. Initial
> > > warning
> > > > >>> stands, that I will start locking down what can get into 1.9.0 at
> > > that
> > > > >>> point.
> > > > >>>
> > > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> > bdbruin@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> No vote indeed, just to gather feedback on a particular fixed
> > point
> > > in
> > > > >>>> time. It also gives a bit more trust to a tarball than to a git
> > > pull.
> > > > >>>>
> > > > >>>> Bolke
> > > > >>>>
> > > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> criccomini@apache.org
> > >
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>> I can do a beta. Is the process significantly different? IIRC,
> > it's
> > > > >>>>> basically the same, just no vote, right?
> > > > >>>>>
> > > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > > bdbruin@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> Are you sure you want to go ahead and do RCs right away?
> Isn’t a
> > > > >> beta
> > > > >>> a
> > > > >>>>>> bit smarter?
> > > > >>>>>>
> > > > >>>>>> - Bolke
> > > > >>>>>>
> > > > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> > criccomini@apache.org
> > > >
> > > > >>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Hey all,
> > > > >>>>>>>
> > > > >>>>>>> I want to send out a warning that I'm planning to cut the
> > stable
> > > > >>> branch
> > > > >>>>>>> next week, and begin the RC1 release vote. Once the stable
> > branch
> > > > >> is
> > > > >>>>>> cut, I
> > > > >>>>>>> will be locking down what commits get cherry picked into the
> > > > >> branch,
> > > > >>>> and
> > > > >>>>>>> will only be doing PRs that are required to get the release
> > out.
> > > > >>>>>>>
> > > > >>>>>>> Cheers,
> > > > >>>>>>> Chris
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > > >>>> criccomini@apache.org
> > > > >>>>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Hey all,
> > > > >>>>>>>>
> > > > >>>>>>>> An update on the 1.9.0 release. Here are the outstanding PRs
> > > that
> > > > >>> are
> > > > >>>>>>>> slated to be included into 1.9.0:
> > > > >>>>>>>>
> > > > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> > endpoint
> > > > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > > > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > > > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded
> code
> > > > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > > > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> > > > >> exception
> > > > >>>> for
> > > > >>>>>>>> @on
> > > > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include
> > paused
> > > > >>> DAGs
> > > > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log
> > to
> > > > >>> stdout
> > > > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> > instances
> > > > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> > exception
> > > > >> for
> > > > >>>>>>>> @once
> > > > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes it
> > to
> > > > >> fail
> > > > >>>>>>>> AIRFLOW-914  |Open      |Refactor
> > BackfillJobTest.test_backfill_
> > > > >>>>>> examples
> > > > >>>>>>>> to
> > > > >>>>>>>> AIRFLOW-913  |Open      |Refactor
> > tests.CoreTest.test_scheduler_
> > > > >> job
> > > > >>>> to
> > > > >>>>>>>> real
> > > > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > > > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs by
> > > > >> default
> > > > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > > > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive
> > conf
> > > > >>>>>>>>
> > > > >>>>>>>> I will be locking down what can get cherry-picked into the
> > 1.9.0
> > > > >>>> branch
> > > > >>>>>>>> shortly, so if you have something you want in, please set
> the
> > > fix
> > > > >>>>>> version
> > > > >>>>>>>> to 1.9.0.
> > > > >>>>>>>>
> > > > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and
> it
> > > has
> > > > >>>> been
> > > > >>>>>>>> running smoothly for several days.
> > > > >>>>>>>>
> > > > >>>>>>>> ** I could really use help verifying stability. If you run
> > > > >> Airflow,
> > > > >>>> it's
> > > > >>>>>>>> in your best interest to deploy the 1.9.0 test branch
> > somewhere,
> > > > >> and
> > > > >>>>>> verify
> > > > >>>>>>>> it's working for your workload. **
> > > > >>>>>>>>
> > > > >>>>>>>> Cheers,
> > > > >>>>>>>> Chris
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Ryan Buckley <ry...@bluecore.com>.

Would it be possible to include AIRFLOW-1587?
Running dags from the UI is currently broken on the 1.9.0 branch due to
this issue.

https://github.com/apache/incubator-airflow/pull/2590

Thanks,
Ryan

On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:

> Hi All,
>
> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> https://github.com/apache/incubator-airflow/pull/2631
>
> Currently importing a custom logging configuration is not work (as far as I
> know). Any feedback on the PR would also be appreciated.
>
> Cheers, Fokko
>
>
> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:
>
> > Done!
> >
> > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> > michael.crawford@modernizingmedicine.com> wrote:
> >
> > > Can you slide the aws and emr connection type fix in?
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > > https://github.com/apache/incubator-airflow/pull/2626 <
> > > https://github.com/apache/incubator-airflow/pull/2626>
> > >
> > > It keeps the connection type from getting blanked out on edit for these
> > > types.
> > >
> > > Thanks,
> > > Mike
> > >
> > >
> > >
> > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <cr...@apache.org>
> > > wrote:
> > > >
> > > > Absolutely. Just cherry-picked. I've been looking forward to these
> > fixes!
> > > >
> > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <alex.guziel@airbnb.com
> .
> > > invalid
> > > >> wrote:
> > > >
> > > >> Can we get this in?
> > > >>
> > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > > >>
> > > >> https://github.com/apache/incubator-airflow/commit/
> > > >> b6d2e0a46978e93e16576604624f57d1388814f2
> > > >> https://github.com/apache/incubator-airflow/commit/
> > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > > >>
> > > >> It speeds up loading times a lot, so it's a good thing to have in
> 1.9.
> > > >>
> > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > > criccomini@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Sounds good. I'll plan on stable+beta next week, then. Initial
> > warning
> > > >>> stands, that I will start locking down what can get into 1.9.0 at
> > that
> > > >>> point.
> > > >>>
> > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
> bdbruin@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> No vote indeed, just to gather feedback on a particular fixed
> point
> > in
> > > >>>> time. It also gives a bit more trust to a tarball than to a git
> > pull.
> > > >>>>
> > > >>>> Bolke
> > > >>>>
> > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <criccomini@apache.org
> >
> > > >>> wrote:
> > > >>>>>
> > > >>>>> I can do a beta. Is the process significantly different? IIRC,
> it's
> > > >>>>> basically the same, just no vote, right?
> > > >>>>>
> > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> > bdbruin@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Are you sure you want to go ahead and do RCs right away? Isn’t a
> > > >> beta
> > > >>> a
> > > >>>>>> bit smarter?
> > > >>>>>>
> > > >>>>>> - Bolke
> > > >>>>>>
> > > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> criccomini@apache.org
> > >
> > > >>>> wrote:
> > > >>>>>>>
> > > >>>>>>> Hey all,
> > > >>>>>>>
> > > >>>>>>> I want to send out a warning that I'm planning to cut the
> stable
> > > >>> branch
> > > >>>>>>> next week, and begin the RC1 release vote. Once the stable
> branch
> > > >> is
> > > >>>>>> cut, I
> > > >>>>>>> will be locking down what commits get cherry picked into the
> > > >> branch,
> > > >>>> and
> > > >>>>>>> will only be doing PRs that are required to get the release
> out.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Chris
> > > >>>>>>>
> > > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > >>>> criccomini@apache.org
> > > >>>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hey all,
> > > >>>>>>>>
> > > >>>>>>>> An update on the 1.9.0 release. Here are the outstanding PRs
> > that
> > > >>> are
> > > >>>>>>>> slated to be included into 1.9.0:
> > > >>>>>>>>
> > > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable
> endpoint
> > > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> > > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> > > >> exception
> > > >>>> for
> > > >>>>>>>> @on
> > > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include
> paused
> > > >>> DAGs
> > > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log
> to
> > > >>> stdout
> > > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task
> instances
> > > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
> exception
> > > >> for
> > > >>>>>>>> @once
> > > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes it
> to
> > > >> fail
> > > >>>>>>>> AIRFLOW-914  |Open      |Refactor
> BackfillJobTest.test_backfill_
> > > >>>>>> examples
> > > >>>>>>>> to
> > > >>>>>>>> AIRFLOW-913  |Open      |Refactor
> tests.CoreTest.test_scheduler_
> > > >> job
> > > >>>> to
> > > >>>>>>>> real
> > > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs by
> > > >> default
> > > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive
> conf
> > > >>>>>>>>
> > > >>>>>>>> I will be locking down what can get cherry-picked into the
> 1.9.0
> > > >>>> branch
> > > >>>>>>>> shortly, so if you have something you want in, please set the
> > fix
> > > >>>>>> version
> > > >>>>>>>> to 1.9.0.
> > > >>>>>>>>
> > > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it
> > has
> > > >>>> been
> > > >>>>>>>> running smoothly for several days.
> > > >>>>>>>>
> > > >>>>>>>> ** I could really use help verifying stability. If you run
> > > >> Airflow,
> > > >>>> it's
> > > >>>>>>>> in your best interest to deploy the 1.9.0 test branch
> somewhere,
> > > >> and
> > > >>>>>> verify
> > > >>>>>>>> it's working for your workload. **
> > > >>>>>>>>
> > > >>>>>>>> Cheers,
> > > >>>>>>>> Chris
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.

Hi All,

I would like to include AIRFLOW-1611 in the 1.9.0 release:
https://github.com/apache/incubator-airflow/pull/2631

Currently importing a custom logging configuration is not work (as far as I
know). Any feedback on the PR would also be appreciated.

Cheers, Fokko


2017-09-25 23:27 GMT+02:00 Chris Riccomini <cr...@apache.org>:

> Done!
>
> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> michael.crawford@modernizingmedicine.com> wrote:
>
> > Can you slide the aws and emr connection type fix in?
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> > https://issues.apache.org/jira/browse/AIRFLOW-1636>
> > https://github.com/apache/incubator-airflow/pull/2626 <
> > https://github.com/apache/incubator-airflow/pull/2626>
> >
> > It keeps the connection type from getting blanked out on edit for these
> > types.
> >
> > Thanks,
> > Mike
> >
> >
> >
> > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> > >
> > > Absolutely. Just cherry-picked. I've been looking forward to these
> fixes!
> > >
> > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <alex.guziel@airbnb.com.
> > invalid
> > >> wrote:
> > >
> > >> Can we get this in?
> > >>
> > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> > >>
> > >> https://github.com/apache/incubator-airflow/commit/
> > >> b6d2e0a46978e93e16576604624f57d1388814f2
> > >> https://github.com/apache/incubator-airflow/commit/
> > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> > >>
> > >> It speeds up loading times a lot, so it's a good thing to have in 1.9.
> > >>
> > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> > criccomini@apache.org>
> > >> wrote:
> > >>
> > >>> Sounds good. I'll plan on stable+beta next week, then. Initial
> warning
> > >>> stands, that I will start locking down what can get into 1.9.0 at
> that
> > >>> point.
> > >>>
> > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> No vote indeed, just to gather feedback on a particular fixed point
> in
> > >>>> time. It also gives a bit more trust to a tarball than to a git
> pull.
> > >>>>
> > >>>> Bolke
> > >>>>
> > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org>
> > >>> wrote:
> > >>>>>
> > >>>>> I can do a beta. Is the process significantly different? IIRC, it's
> > >>>>> basically the same, just no vote, right?
> > >>>>>
> > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
> bdbruin@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Are you sure you want to go ahead and do RCs right away? Isn’t a
> > >> beta
> > >>> a
> > >>>>>> bit smarter?
> > >>>>>>
> > >>>>>> - Bolke
> > >>>>>>
> > >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <criccomini@apache.org
> >
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>> Hey all,
> > >>>>>>>
> > >>>>>>> I want to send out a warning that I'm planning to cut the stable
> > >>> branch
> > >>>>>>> next week, and begin the RC1 release vote. Once the stable branch
> > >> is
> > >>>>>> cut, I
> > >>>>>>> will be locking down what commits get cherry picked into the
> > >> branch,
> > >>>> and
> > >>>>>>> will only be doing PRs that are required to get the release out.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Chris
> > >>>>>>>
> > >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > >>>> criccomini@apache.org
> > >>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hey all,
> > >>>>>>>>
> > >>>>>>>> An update on the 1.9.0 release. Here are the outstanding PRs
> that
> > >>> are
> > >>>>>>>> slated to be included into 1.9.0:
> > >>>>>>>>
> > >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> > >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> > >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> > >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> > >> exception
> > >>>> for
> > >>>>>>>> @on
> > >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused
> > >>> DAGs
> > >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to
> > >>> stdout
> > >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> > >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception
> > >> for
> > >>>>>>>> @once
> > >>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes it to
> > >> fail
> > >>>>>>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> > >>>>>> examples
> > >>>>>>>> to
> > >>>>>>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_
> > >> job
> > >>>> to
> > >>>>>>>> real
> > >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs by
> > >> default
> > >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> > >>>>>>>>
> > >>>>>>>> I will be locking down what can get cherry-picked into the 1.9.0
> > >>>> branch
> > >>>>>>>> shortly, so if you have something you want in, please set the
> fix
> > >>>>>> version
> > >>>>>>>> to 1.9.0.
> > >>>>>>>>
> > >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it
> has
> > >>>> been
> > >>>>>>>> running smoothly for several days.
> > >>>>>>>>
> > >>>>>>>> ** I could really use help verifying stability. If you run
> > >> Airflow,
> > >>>> it's
> > >>>>>>>> in your best interest to deploy the 1.9.0 test branch somewhere,
> > >> and
> > >>>>>> verify
> > >>>>>>>> it's working for your workload. **
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> Chris
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Done!

On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
michael.crawford@modernizingmedicine.com> wrote:

> Can you slide the aws and emr connection type fix in?
>
> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> https://github.com/apache/incubator-airflow/pull/2626 <
> https://github.com/apache/incubator-airflow/pull/2626>
>
> It keeps the connection type from getting blanked out on edit for these
> types.
>
> Thanks,
> Mike
>
>
>
> > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <cr...@apache.org>
> wrote:
> >
> > Absolutely. Just cherry-picked. I've been looking forward to these fixes!
> >
> > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <alex.guziel@airbnb.com.
> invalid
> >> wrote:
> >
> >> Can we get this in?
> >>
> >> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>
> >> https://github.com/apache/incubator-airflow/commit/
> >> b6d2e0a46978e93e16576604624f57d1388814f2
> >> https://github.com/apache/incubator-airflow/commit/
> >> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>
> >> It speeds up loading times a lot, so it's a good thing to have in 1.9.
> >>
> >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
> criccomini@apache.org>
> >> wrote:
> >>
> >>> Sounds good. I'll plan on stable+beta next week, then. Initial warning
> >>> stands, that I will start locking down what can get into 1.9.0 at that
> >>> point.
> >>>
> >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com>
> >>> wrote:
> >>>
> >>>> No vote indeed, just to gather feedback on a particular fixed point in
> >>>> time. It also gives a bit more trust to a tarball than to a git pull.
> >>>>
> >>>> Bolke
> >>>>
> >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org>
> >>> wrote:
> >>>>>
> >>>>> I can do a beta. Is the process significantly different? IIRC, it's
> >>>>> basically the same, just no vote, right?
> >>>>>
> >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Are you sure you want to go ahead and do RCs right away? Isn’t a
> >> beta
> >>> a
> >>>>>> bit smarter?
> >>>>>>
> >>>>>> - Bolke
> >>>>>>
> >>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Hey all,
> >>>>>>>
> >>>>>>> I want to send out a warning that I'm planning to cut the stable
> >>> branch
> >>>>>>> next week, and begin the RC1 release vote. Once the stable branch
> >> is
> >>>>>> cut, I
> >>>>>>> will be locking down what commits get cherry picked into the
> >> branch,
> >>>> and
> >>>>>>> will only be doing PRs that are required to get the release out.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Chris
> >>>>>>>
> >>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hey all,
> >>>>>>>>
> >>>>>>>> An update on the 1.9.0 release. Here are the outstanding PRs that
> >>> are
> >>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>
> >>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> >>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> >>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> >>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> >>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> >>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> >> exception
> >>>> for
> >>>>>>>> @on
> >>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused
> >>> DAGs
> >>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to
> >>> stdout
> >>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> >>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception
> >> for
> >>>>>>>> @once
> >>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes it to
> >> fail
> >>>>>>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> >>>>>> examples
> >>>>>>>> to
> >>>>>>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_
> >> job
> >>>> to
> >>>>>>>> real
> >>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> >>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs by
> >> default
> >>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> >>>>>>>>
> >>>>>>>> I will be locking down what can get cherry-picked into the 1.9.0
> >>>> branch
> >>>>>>>> shortly, so if you have something you want in, please set the fix
> >>>>>> version
> >>>>>>>> to 1.9.0.
> >>>>>>>>
> >>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has
> >>>> been
> >>>>>>>> running smoothly for several days.
> >>>>>>>>
> >>>>>>>> ** I could really use help verifying stability. If you run
> >> Airflow,
> >>>> it's
> >>>>>>>> in your best interest to deploy the 1.9.0 test branch somewhere,
> >> and
> >>>>>> verify
> >>>>>>>> it's working for your workload. **
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Michael Crawford <mi...@modernizingmedicine.com>.

Can you slide the aws and emr connection type fix in?

https://issues.apache.org/jira/browse/AIRFLOW-1636 <https://issues.apache.org/jira/browse/AIRFLOW-1636>
https://github.com/apache/incubator-airflow/pull/2626 <https://github.com/apache/incubator-airflow/pull/2626>

It keeps the connection type from getting blanked out on edit for these types.

Thanks,
Mike



> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <cr...@apache.org> wrote:
> 
> Absolutely. Just cherry-picked. I've been looking forward to these fixes!
> 
> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <alex.guziel@airbnb.com.invalid
>> wrote:
> 
>> Can we get this in?
>> 
>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> 
>> https://github.com/apache/incubator-airflow/commit/
>> b6d2e0a46978e93e16576604624f57d1388814f2
>> https://github.com/apache/incubator-airflow/commit/
>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> 
>> It speeds up loading times a lot, so it's a good thing to have in 1.9.
>> 
>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <cr...@apache.org>
>> wrote:
>> 
>>> Sounds good. I'll plan on stable+beta next week, then. Initial warning
>>> stands, that I will start locking down what can get into 1.9.0 at that
>>> point.
>>> 
>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>> 
>>>> No vote indeed, just to gather feedback on a particular fixed point in
>>>> time. It also gives a bit more trust to a tarball than to a git pull.
>>>> 
>>>> Bolke
>>>> 
>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org>
>>> wrote:
>>>>> 
>>>>> I can do a beta. Is the process significantly different? IIRC, it's
>>>>> basically the same, just no vote, right?
>>>>> 
>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> Are you sure you want to go ahead and do RCs right away? Isn’t a
>> beta
>>> a
>>>>>> bit smarter?
>>>>>> 
>>>>>> - Bolke
>>>>>> 
>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> I want to send out a warning that I'm planning to cut the stable
>>> branch
>>>>>>> next week, and begin the RC1 release vote. Once the stable branch
>> is
>>>>>> cut, I
>>>>>>> will be locking down what commits get cherry picked into the
>> branch,
>>>> and
>>>>>>> will only be doing PRs that are required to get the release out.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
>>>> criccomini@apache.org
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> An update on the 1.9.0 release. Here are the outstanding PRs that
>>> are
>>>>>>>> slated to be included into 1.9.0:
>>>>>>>> 
>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
>> exception
>>>> for
>>>>>>>> @on
>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused
>>> DAGs
>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to
>>> stdout
>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception
>> for
>>>>>>>> @once
>>>>>>>> AIRFLOW-976  |Open      |Mark success running task causes it to
>> fail
>>>>>>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
>>>>>> examples
>>>>>>>> to
>>>>>>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_
>> job
>>>> to
>>>>>>>> real
>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
>>>>>>>> AIRFLOW-888  |Open      |Operators should not push XComs by
>> default
>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
>>>>>>>> 
>>>>>>>> I will be locking down what can get cherry-picked into the 1.9.0
>>>> branch
>>>>>>>> shortly, so if you have something you want in, please set the fix
>>>>>> version
>>>>>>>> to 1.9.0.
>>>>>>>> 
>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has
>>>> been
>>>>>>>> running smoothly for several days.
>>>>>>>> 
>>>>>>>> ** I could really use help verifying stability. If you run
>> Airflow,
>>>> it's
>>>>>>>> in your best interest to deploy the 1.9.0 test branch somewhere,
>> and
>>>>>> verify
>>>>>>>> it's working for your workload. **
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Absolutely. Just cherry-picked. I've been looking forward to these fixes!

On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <alex.guziel@airbnb.com.invalid
> wrote:

> Can we get this in?
>
> https://issues.apache.org/jira/browse/AIRFLOW-1519
> https://issues.apache.org/jira/browse/AIRFLOW-1621
>
> https://github.com/apache/incubator-airflow/commit/
> b6d2e0a46978e93e16576604624f57d1388814f2
> https://github.com/apache/incubator-airflow/commit/
> 656d045e90bf67ca484a3778b2a07a419bfb324a
>
> It speeds up loading times a lot, so it's a good thing to have in 1.9.
>
> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Sounds good. I'll plan on stable+beta next week, then. Initial warning
> > stands, that I will start locking down what can get into 1.9.0 at that
> > point.
> >
> > On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> >
> > > No vote indeed, just to gather feedback on a particular fixed point in
> > > time. It also gives a bit more trust to a tarball than to a git pull.
> > >
> > > Bolke
> > >
> > > > On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org>
> > wrote:
> > > >
> > > > I can do a beta. Is the process significantly different? IIRC, it's
> > > > basically the same, just no vote, right?
> > > >
> > > > On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > >
> > > >> Are you sure you want to go ahead and do RCs right away? Isn’t a
> beta
> > a
> > > >> bit smarter?
> > > >>
> > > >> - Bolke
> > > >>
> > > >>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org>
> > > wrote:
> > > >>>
> > > >>> Hey all,
> > > >>>
> > > >>> I want to send out a warning that I'm planning to cut the stable
> > branch
> > > >>> next week, and begin the RC1 release vote. Once the stable branch
> is
> > > >> cut, I
> > > >>> will be locking down what commits get cherry picked into the
> branch,
> > > and
> > > >>> will only be doing PRs that are required to get the release out.
> > > >>>
> > > >>> Cheers,
> > > >>> Chris
> > > >>>
> > > >>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > > criccomini@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>>> Hey all,
> > > >>>>
> > > >>>> An update on the 1.9.0 release. Here are the outstanding PRs that
> > are
> > > >>>> slated to be included into 1.9.0:
> > > >>>>
> > > >>>> ISSUE ID     |STATUS    |DESCRIPTION
> > > >>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> > > >>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > > >>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > > >>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > > >>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > > >>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> > > >>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > > >>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run()
> exception
> > > for
> > > >>>> @on
> > > >>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused
> > DAGs
> > > >>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to
> > stdout
> > > >>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> > > >>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception
> for
> > > >>>> @once
> > > >>>> AIRFLOW-976  |Open      |Mark success running task causes it to
> fail
> > > >>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> > > >> examples
> > > >>>> to
> > > >>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_
> job
> > > to
> > > >>>> real
> > > >>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > > >>>> AIRFLOW-888  |Open      |Operators should not push XComs by
> default
> > > >>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > > >>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > > >>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> > > >>>>
> > > >>>> I will be locking down what can get cherry-picked into the 1.9.0
> > > branch
> > > >>>> shortly, so if you have something you want in, please set the fix
> > > >> version
> > > >>>> to 1.9.0.
> > > >>>>
> > > >>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has
> > > been
> > > >>>> running smoothly for several days.
> > > >>>>
> > > >>>> ** I could really use help verifying stability. If you run
> Airflow,
> > > it's
> > > >>>> in your best interest to deploy the 1.9.0 test branch somewhere,
> and
> > > >> verify
> > > >>>> it's working for your workload. **
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Chris
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: Airflow 1.9.0 status

Posted by Alex Guziel <al...@airbnb.com.INVALID>.

Can we get this in?

https://issues.apache.org/jira/browse/AIRFLOW-1519
https://issues.apache.org/jira/browse/AIRFLOW-1621

https://github.com/apache/incubator-airflow/commit/b6d2e0a46978e93e16576604624f57d1388814f2
https://github.com/apache/incubator-airflow/commit/656d045e90bf67ca484a3778b2a07a419bfb324a

It speeds up loading times a lot, so it's a good thing to have in 1.9.

On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Sounds good. I'll plan on stable+beta next week, then. Initial warning
> stands, that I will start locking down what can get into 1.9.0 at that
> point.
>
> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
>
> > No vote indeed, just to gather feedback on a particular fixed point in
> > time. It also gives a bit more trust to a tarball than to a git pull.
> >
> > Bolke
> >
> > > On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org>
> wrote:
> > >
> > > I can do a beta. Is the process significantly different? IIRC, it's
> > > basically the same, just no vote, right?
> > >
> > > On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > >> Are you sure you want to go ahead and do RCs right away? Isn’t a beta
> a
> > >> bit smarter?
> > >>
> > >> - Bolke
> > >>
> > >>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org>
> > wrote:
> > >>>
> > >>> Hey all,
> > >>>
> > >>> I want to send out a warning that I'm planning to cut the stable
> branch
> > >>> next week, and begin the RC1 release vote. Once the stable branch is
> > >> cut, I
> > >>> will be locking down what commits get cherry picked into the branch,
> > and
> > >>> will only be doing PRs that are required to get the release out.
> > >>>
> > >>> Cheers,
> > >>> Chris
> > >>>
> > >>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> > criccomini@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hey all,
> > >>>>
> > >>>> An update on the 1.9.0 release. Here are the outstanding PRs that
> are
> > >>>> slated to be included into 1.9.0:
> > >>>>
> > >>>> ISSUE ID     |STATUS    |DESCRIPTION
> > >>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> > >>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> > >>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> > >>>> AIRFLOW-1604 |Open      |Rename the logger to log
> > >>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> > >>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> > >>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> > >>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception
> > for
> > >>>> @on
> > >>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused
> DAGs
> > >>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to
> stdout
> > >>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> > >>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
> > >>>> @once
> > >>>> AIRFLOW-976  |Open      |Mark success running task causes it to fail
> > >>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> > >> examples
> > >>>> to
> > >>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job
> > to
> > >>>> real
> > >>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> > >>>> AIRFLOW-888  |Open      |Operators should not push XComs by default
> > >>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> > >>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> > >>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> > >>>>
> > >>>> I will be locking down what can get cherry-picked into the 1.9.0
> > branch
> > >>>> shortly, so if you have something you want in, please set the fix
> > >> version
> > >>>> to 1.9.0.
> > >>>>
> > >>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has
> > been
> > >>>> running smoothly for several days.
> > >>>>
> > >>>> ** I could really use help verifying stability. If you run Airflow,
> > it's
> > >>>> in your best interest to deploy the 1.9.0 test branch somewhere, and
> > >> verify
> > >>>> it's working for your workload. **
> > >>>>
> > >>>> Cheers,
> > >>>> Chris
> > >>>>
> > >>
> > >>
> >
> >
>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Sounds good. I'll plan on stable+beta next week, then. Initial warning
stands, that I will start locking down what can get into 1.9.0 at that
point.

On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> No vote indeed, just to gather feedback on a particular fixed point in
> time. It also gives a bit more trust to a tarball than to a git pull.
>
> Bolke
>
> > On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org> wrote:
> >
> > I can do a beta. Is the process significantly different? IIRC, it's
> > basically the same, just no vote, right?
> >
> > On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> >> Are you sure you want to go ahead and do RCs right away? Isn’t a beta a
> >> bit smarter?
> >>
> >> - Bolke
> >>
> >>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org>
> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> I want to send out a warning that I'm planning to cut the stable branch
> >>> next week, and begin the RC1 release vote. Once the stable branch is
> >> cut, I
> >>> will be locking down what commits get cherry picked into the branch,
> and
> >>> will only be doing PRs that are required to get the release out.
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <
> criccomini@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> An update on the 1.9.0 release. Here are the outstanding PRs that are
> >>>> slated to be included into 1.9.0:
> >>>>
> >>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> >>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
> >>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> >>>> AIRFLOW-1604 |Open      |Rename the logger to log
> >>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> >>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> >>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> >>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception
> for
> >>>> @on
> >>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
> >>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
> >>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> >>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
> >>>> @once
> >>>> AIRFLOW-976  |Open      |Mark success running task causes it to fail
> >>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> >> examples
> >>>> to
> >>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job
> to
> >>>> real
> >>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
> >>>> AIRFLOW-888  |Open      |Operators should not push XComs by default
> >>>> AIRFLOW-828  |Open      |Add maximum size for XComs
> >>>> AIRFLOW-825  |Open      |Add Dataflow semantics
> >>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> >>>>
> >>>> I will be locking down what can get cherry-picked into the 1.9.0
> branch
> >>>> shortly, so if you have something you want in, please set the fix
> >> version
> >>>> to 1.9.0.
> >>>>
> >>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has
> been
> >>>> running smoothly for several days.
> >>>>
> >>>> ** I could really use help verifying stability. If you run Airflow,
> it's
> >>>> in your best interest to deploy the 1.9.0 test branch somewhere, and
> >> verify
> >>>> it's working for your workload. **
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

No vote indeed, just to gather feedback on a particular fixed point in time. It also gives a bit more trust to a tarball than to a git pull.

Bolke

> On 20 Sep 2017, at 20:09, Chris Riccomini <cr...@apache.org> wrote:
> 
> I can do a beta. Is the process significantly different? IIRC, it's
> basically the same, just no vote, right?
> 
> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> 
>> Are you sure you want to go ahead and do RCs right away? Isn’t a beta a
>> bit smarter?
>> 
>> - Bolke
>> 
>>> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org> wrote:
>>> 
>>> Hey all,
>>> 
>>> I want to send out a warning that I'm planning to cut the stable branch
>>> next week, and begin the RC1 release vote. Once the stable branch is
>> cut, I
>>> will be locking down what commits get cherry picked into the branch, and
>>> will only be doing PRs that are required to get the release out.
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <criccomini@apache.org
>>> 
>>> wrote:
>>> 
>>>> Hey all,
>>>> 
>>>> An update on the 1.9.0 release. Here are the outstanding PRs that are
>>>> slated to be included into 1.9.0:
>>>> 
>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
>>>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>>>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception for
>>>> @on
>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
>>>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
>>>> @once
>>>> AIRFLOW-976  |Open      |Mark success running task causes it to fail
>>>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
>> examples
>>>> to
>>>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job to
>>>> real
>>>> AIRFLOW-912  |Open      |Refactor tests and build matrix
>>>> AIRFLOW-888  |Open      |Operators should not push XComs by default
>>>> AIRFLOW-828  |Open      |Add maximum size for XComs
>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
>>>> 
>>>> I will be locking down what can get cherry-picked into the 1.9.0 branch
>>>> shortly, so if you have something you want in, please set the fix
>> version
>>>> to 1.9.0.
>>>> 
>>>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has been
>>>> running smoothly for several days.
>>>> 
>>>> ** I could really use help verifying stability. If you run Airflow, it's
>>>> in your best interest to deploy the 1.9.0 test branch somewhere, and
>> verify
>>>> it's working for your workload. **
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>> 
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

I can do a beta. Is the process significantly different? IIRC, it's
basically the same, just no vote, right?

On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> Are you sure you want to go ahead and do RCs right away? Isn’t a beta a
> bit smarter?
>
> - Bolke
>
> > On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org> wrote:
> >
> > Hey all,
> >
> > I want to send out a warning that I'm planning to cut the stable branch
> > next week, and begin the RC1 release vote. Once the stable branch is
> cut, I
> > will be locking down what commits get cherry picked into the branch, and
> > will only be doing PRs that are required to get the release out.
> >
> > Cheers,
> > Chris
> >
> > On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> Hey all,
> >>
> >> An update on the 1.9.0 release. Here are the outstanding PRs that are
> >> slated to be included into 1.9.0:
> >>
> >> ISSUE ID     |STATUS    |DESCRIPTION
> >> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> >> AIRFLOW-1611 |Open      |Customize logging in Airflow
> >> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> >> AIRFLOW-1604 |Open      |Rename the logger to log
> >> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> >> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> >> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> >> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception for
> >> @on
> >> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
> >> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
> >> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> >> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
> >> @once
> >> AIRFLOW-976  |Open      |Mark success running task causes it to fail
> >> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_
> examples
> >> to
> >> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job to
> >> real
> >> AIRFLOW-912  |Open      |Refactor tests and build matrix
> >> AIRFLOW-888  |Open      |Operators should not push XComs by default
> >> AIRFLOW-828  |Open      |Add maximum size for XComs
> >> AIRFLOW-825  |Open      |Add Dataflow semantics
> >> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
> >>
> >> I will be locking down what can get cherry-picked into the 1.9.0 branch
> >> shortly, so if you have something you want in, please set the fix
> version
> >> to 1.9.0.
> >>
> >> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has been
> >> running smoothly for several days.
> >>
> >> ** I could really use help verifying stability. If you run Airflow, it's
> >> in your best interest to deploy the 1.9.0 test branch somewhere, and
> verify
> >> it's working for your workload. **
> >>
> >> Cheers,
> >> Chris
> >>
>
>

Re: Airflow 1.9.0 status

Posted by Bolke de Bruin <bd...@gmail.com>.

Are you sure you want to go ahead and do RCs right away? Isn’t a beta a bit smarter?

- Bolke

> On 20 Sep 2017, at 19:41, Chris Riccomini <cr...@apache.org> wrote:
> 
> Hey all,
> 
> I want to send out a warning that I'm planning to cut the stable branch
> next week, and begin the RC1 release vote. Once the stable branch is cut, I
> will be locking down what commits get cherry picked into the branch, and
> will only be doing PRs that are required to get the release out.
> 
> Cheers,
> Chris
> 
> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <cr...@apache.org>
> wrote:
> 
>> Hey all,
>> 
>> An update on the 1.9.0 release. Here are the outstanding PRs that are
>> slated to be included into 1.9.0:
>> 
>> ISSUE ID     |STATUS    |DESCRIPTION
>> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
>> AIRFLOW-1611 |Open      |Customize logging in Airflow
>> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
>> AIRFLOW-1604 |Open      |Rename the logger to log
>> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
>> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
>> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception for
>> @on
>> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
>> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
>> AIRFLOW-1015 |Open      |TreeView displayed over task instances
>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
>> @once
>> AIRFLOW-976  |Open      |Mark success running task causes it to fail
>> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_examples
>> to
>> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job to
>> real
>> AIRFLOW-912  |Open      |Refactor tests and build matrix
>> AIRFLOW-888  |Open      |Operators should not push XComs by default
>> AIRFLOW-828  |Open      |Add maximum size for XComs
>> AIRFLOW-825  |Open      |Add Dataflow semantics
>> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
>> 
>> I will be locking down what can get cherry-picked into the 1.9.0 branch
>> shortly, so if you have something you want in, please set the fix version
>> to 1.9.0.
>> 
>> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has been
>> running smoothly for several days.
>> 
>> ** I could really use help verifying stability. If you run Airflow, it's
>> in your best interest to deploy the 1.9.0 test branch somewhere, and verify
>> it's working for your workload. **
>> 
>> Cheers,
>> Chris
>>

Re: Airflow 1.9.0 status

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

I want to send out a warning that I'm planning to cut the stable branch
next week, and begin the RC1 release vote. Once the stable branch is cut, I
will be locking down what commits get cherry picked into the branch, and
will only be doing PRs that are required to get the release out.

Cheers,
Chris

On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Hey all,
>
> An update on the 1.9.0 release. Here are the outstanding PRs that are
> slated to be included into 1.9.0:
>
> ISSUE ID     |STATUS    |DESCRIPTION
> AIRFLOW-1617 |Open      |XSS Vulnerability in Variable endpoint
> AIRFLOW-1611 |Open      |Customize logging in Airflow
> AIRFLOW-1605 |Reopened  |Fix log source of local loggers
> AIRFLOW-1604 |Open      |Rename the logger to log
> AIRFLOW-1525 |Open      |Fix minor LICENSE & NOTICE issue
> AIRFLOW-1499 |In Progres|Eliminate duplicate and unneeded code
> AIRFLOW-1198 |Open      |HDFSOperator to operate HDFS
> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_run() exception for
> @on
> AIRFLOW-1019 |Open      |active_dagruns shouldn't include paused DAGs
> AIRFLOW-1018 |Open      |Scheduler DAG processes can not log to stdout
> AIRFLOW-1015 |Open      |TreeView displayed over task instances
> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas() exception for
> @once
> AIRFLOW-976  |Open      |Mark success running task causes it to fail
> AIRFLOW-914  |Open      |Refactor BackfillJobTest.test_backfill_examples
> to
> AIRFLOW-913  |Open      |Refactor tests.CoreTest.test_scheduler_job to
> real
> AIRFLOW-912  |Open      |Refactor tests and build matrix
> AIRFLOW-888  |Open      |Operators should not push XComs by default
> AIRFLOW-828  |Open      |Add maximum size for XComs
> AIRFLOW-825  |Open      |Add Dataflow semantics
> AIRFLOW-788  |Open      |Context unexpectedly added to hive conf
>
> I will be locking down what can get cherry-picked into the 1.9.0 branch
> shortly, so if you have something you want in, please set the fix version
> to 1.9.0.
>
> We (at WePay) have deployed 1.9.0 into our dev cluster, and it has been
> running smoothly for several days.
>
> ** I could really use help verifying stability. If you run Airflow, it's
> in your best interest to deploy the 1.9.0 test branch somewhere, and verify
> it's working for your workload. **
>
> Cheers,
> Chris
>