You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Deng Xiaodong <xd...@gmail.com> on 2018/09/03 12:47:24 UTC

Re: Call for fixes for Airflow 1.10.1

Hi Ash,

JIRA ticket 2922 <https://issues.apache.org/jira/browse/AIRFLOW-2922> was
fixed together with *2921* in PR 3773
<https://github.com/apache/incubator-airflow/pull/3773> (which was already
listed by you). Understand that it may be included in 1.10.1 automatically
if this PR is included, but may still be good to update ticket 2922's "*Fix
Version*" to "*1.10.1*" in JIRA for better tracking?

Thanks!

Best regards,
XD

On Mon, Sep 3, 2018 at 8:20 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Hi everyone,
>
> I'm starting the process of gathering fixes for a 1.10.1. So far the list
> of issues I have that we should pull in are
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AIRFLOW%20AND%20fixVersion%20%3D%201.10.1%20ORDER%20BY%20key%20ASC
> (reproduces below)
>
> I will start pushing these as cherry-picked commits to the v1-10-stable
> branch today.
>
> If you have something that is not in the list below let me know. I'd like
> to keep this to bug fixes against 1.10.0 only if possible.
>
> https://issues.apache.org/jira/browse/AIRFLOW-2145 Deadlock after
> clearing a running task
> https://github.com/apache/incubator-airflow/pull/3657
>
> https://issues.apache.org/jira/browse/AIRFLOW-2476 update tabulate dep to
> 0.8.2
> https://github.com/apache/incubator-airflow/pull/3835
>
> https://issues.apache.org/jira/browse/AIRFLOW-2778 Bad Import in
> collect_dag in DagBag
> https://github.com/apache/incubator-airflow/pull/3624
>
> https://issues.apache.org/jira/browse/AIRFLOW-2900 Show code for packaged
> DAGs
> https://github.com/apache/incubator-airflow/pull/3749
>
> https://issues.apache.org/jira/browse/AIRFLOW-2949 Add syntax highlight
> for single quote strings
> https://github.com/apache/incubator-airflow/pull/3795
>
> https://issues.apache.org/jira/browse/AIRFLOW-2984 Cannot convert
> naive_datetime when task has a naive start_date/end_date
> https://github.com/apache/incubator-airflow/pull/3822
>
> https://issues.apache.org/jira/browse/AIRFLOW-2979 Deprecated Celery
> Option not in Options list
> https://github.com/apache/incubator-airflow/pull/3832
>
> https://issues.apache.org/jira/browse/AIRFLOW-2989 No Parameter to change
> bootDiskType for DataprocClusterCreateOperator
> https://github.com/apache/incubator-airflow/pull/3825
>
> https://issues.apache.org/jira/browse/AIRFLOW-2990 Docstrings for
> Hooks/Operators are in incorrect format
> https://github.com/apache/incubator-airflow/pull/3820
>
> https://issues.apache.org/jira/browse/AIRFLOW-2994 flatten_results in
> BigQueryOperator/BigQueryHook should default to None
> https://github.com/apache/incubator-airflow/pull/3829
>
>
> In addition to those PRs which are already marked with Fix Version of
> 1.10.1 I think we should also pull in these:
>
>
> https://issues.apache.org/jira/browse/AIRFLOW-2713 Rename async variable
> for Python 3.7.0 compatibility
> https://github.com/apache/incubator-airflow/pull/3561
>
> https://issues.apache.org/jira/browse/AIRFLOW-2895 Prevent scheduler from
> spamming heartbeats/logs
> https://github.com/apache/incubator-airflow/pull/3747
>
> https://issues.apache.org/jira/browse/AIRFLOW-2921 A trivial
> incorrectness in CeleryExecutor()
> https://github.com/apache/incubator-airflow/pull/3773
>
> https://issues.apache.org/jira/browse/AIRFLOW-2866 Missing CSRF Token
> Error on Web RBAC UI Create/Update Operations
> https://github.com/apache/incubator-airflow/pull/3804
>
>
> https://issues.apache.org/jira/browse/AIRFLOW-2951
> https://github.com/apache/incubator-airflow/pull/3798 Update dag_run
> table end_date when state change
> (though as written it has a few other deps to cherry pick in, so will see
> about this one)
>
>

Re: Call for fixes for Airflow 1.10.1

Posted by Kaxil Naik <ka...@gmail.com>.
Also please add the following fixes:


   - https://issues.apache.org/jira/browse/AIRFLOW-2763 (Add check to
   validate worker connectivity to metadata)
   - https://issues.apache.org/jira/browse/AIRFLOW-1874 (use_legacy_sql
   added to BigQueryCheck operators)
   - https://issues.apache.org/jira/browse/AIRFLOW-2786 *Definitely this
   one *(Gracefully handle Variable import errors)
   - https://issues.apache.org/jira/browse/AIRFLOW-2860 (DruidHook: time
   check is wrong)
   - https://issues.apache.org/jira/browse/AIRFLOW-2893 (fix stuck dataflow
   job due to name mismatch)
   - https://github.com/apache/incubator-airflow/pull/3757 (Fix Sendgrid
   attachments bytes error)
   - https://issues.apache.org/jira/browse/AIRFLOW-2905 (Fix Dataflow get
   job API endpoint)
   - https://github.com/apache/incubator-airflow/pull/3778 (Fix some
   operator names in the docs)
   - https://github.com/apache/incubator-airflow/pull/3788 (Replaces
   incorrect env var name in INSTALL)
   - https://issues.apache.org/jira/browse/AIRFLOW-2938 (Handle improperly
   formatted extra field in connection)
   - https://issues.apache.org/jira/browse/AIRFLOW-2981 (Fix TypeError in
   dataflow operators)



On Mon, Sep 3, 2018 at 7:56 PM Driesprong, Fokko <fo...@driesprong.frl>
wrote:

> Ash, thanks for picking this up!
>
> Maybe add this one:
> https://github.com/apache/incubator-airflow/pull/3804
> https://issues.apache.org/jira/browse/AIRFLOW-2866
>
> Cheers, Fokko
>
>
> Op ma 3 sep. 2018 om 14:47 schreef Deng Xiaodong <xd...@gmail.com>:
>
> > Hi Ash,
> >
> > JIRA ticket 2922 <https://issues.apache.org/jira/browse/AIRFLOW-2922>
> was
> > fixed together with *2921* in PR 3773
> > <https://github.com/apache/incubator-airflow/pull/3773> (which was
> already
> > listed by you). Understand that it may be included in 1.10.1
> automatically
> > if this PR is included, but may still be good to update ticket 2922's
> "*Fix
> > Version*" to "*1.10.1*" in JIRA for better tracking?
> >
> > Thanks!
> >
> > Best regards,
> > XD
> >
> > On Mon, Sep 3, 2018 at 8:20 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > > Hi everyone,
> > >
> > > I'm starting the process of gathering fixes for a 1.10.1. So far the
> list
> > > of issues I have that we should pull in are
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AIRFLOW%20AND%20fixVersion%20%3D%201.10.1%20ORDER%20BY%20key%20ASC
> > > (reproduces below)
> > >
> > > I will start pushing these as cherry-picked commits to the v1-10-stable
> > > branch today.
> > >
> > > If you have something that is not in the list below let me know. I'd
> like
> > > to keep this to bug fixes against 1.10.0 only if possible.
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2145 Deadlock after
> > > clearing a running task
> > > https://github.com/apache/incubator-airflow/pull/3657
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2476 update tabulate dep
> > to
> > > 0.8.2
> > > https://github.com/apache/incubator-airflow/pull/3835
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2778 Bad Import in
> > > collect_dag in DagBag
> > > https://github.com/apache/incubator-airflow/pull/3624
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2900 Show code for
> > packaged
> > > DAGs
> > > https://github.com/apache/incubator-airflow/pull/3749
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2949 Add syntax
> highlight
> > > for single quote strings
> > > https://github.com/apache/incubator-airflow/pull/3795
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2984 Cannot convert
> > > naive_datetime when task has a naive start_date/end_date
> > > https://github.com/apache/incubator-airflow/pull/3822
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2979 Deprecated Celery
> > > Option not in Options list
> > > https://github.com/apache/incubator-airflow/pull/3832
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2989 No Parameter to
> > change
> > > bootDiskType for DataprocClusterCreateOperator
> > > https://github.com/apache/incubator-airflow/pull/3825
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2990 Docstrings for
> > > Hooks/Operators are in incorrect format
> > > https://github.com/apache/incubator-airflow/pull/3820
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2994 flatten_results in
> > > BigQueryOperator/BigQueryHook should default to None
> > > https://github.com/apache/incubator-airflow/pull/3829
> > >
> > >
> > > In addition to those PRs which are already marked with Fix Version of
> > > 1.10.1 I think we should also pull in these:
> > >
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2713 Rename async
> variable
> > > for Python 3.7.0 compatibility
> > > https://github.com/apache/incubator-airflow/pull/3561
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2895 Prevent scheduler
> > from
> > > spamming heartbeats/logs
> > > https://github.com/apache/incubator-airflow/pull/3747
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2921 A trivial
> > > incorrectness in CeleryExecutor()
> > > https://github.com/apache/incubator-airflow/pull/3773
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2866 Missing CSRF Token
> > > Error on Web RBAC UI Create/Update Operations
> > > https://github.com/apache/incubator-airflow/pull/3804
> > >
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2951
> > > https://github.com/apache/incubator-airflow/pull/3798 Update dag_run
> > > table end_date when state change
> > > (though as written it has a few other deps to cherry pick in, so will
> see
> > > about this one)
> > >
> > >
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Call for fixes for Airflow 1.10.1

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Ash, thanks for picking this up!

Maybe add this one:
https://github.com/apache/incubator-airflow/pull/3804
https://issues.apache.org/jira/browse/AIRFLOW-2866

Cheers, Fokko


Op ma 3 sep. 2018 om 14:47 schreef Deng Xiaodong <xd...@gmail.com>:

> Hi Ash,
>
> JIRA ticket 2922 <https://issues.apache.org/jira/browse/AIRFLOW-2922> was
> fixed together with *2921* in PR 3773
> <https://github.com/apache/incubator-airflow/pull/3773> (which was already
> listed by you). Understand that it may be included in 1.10.1 automatically
> if this PR is included, but may still be good to update ticket 2922's "*Fix
> Version*" to "*1.10.1*" in JIRA for better tracking?
>
> Thanks!
>
> Best regards,
> XD
>
> On Mon, Sep 3, 2018 at 8:20 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Hi everyone,
> >
> > I'm starting the process of gathering fixes for a 1.10.1. So far the list
> > of issues I have that we should pull in are
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AIRFLOW%20AND%20fixVersion%20%3D%201.10.1%20ORDER%20BY%20key%20ASC
> > (reproduces below)
> >
> > I will start pushing these as cherry-picked commits to the v1-10-stable
> > branch today.
> >
> > If you have something that is not in the list below let me know. I'd like
> > to keep this to bug fixes against 1.10.0 only if possible.
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2145 Deadlock after
> > clearing a running task
> > https://github.com/apache/incubator-airflow/pull/3657
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2476 update tabulate dep
> to
> > 0.8.2
> > https://github.com/apache/incubator-airflow/pull/3835
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2778 Bad Import in
> > collect_dag in DagBag
> > https://github.com/apache/incubator-airflow/pull/3624
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2900 Show code for
> packaged
> > DAGs
> > https://github.com/apache/incubator-airflow/pull/3749
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2949 Add syntax highlight
> > for single quote strings
> > https://github.com/apache/incubator-airflow/pull/3795
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2984 Cannot convert
> > naive_datetime when task has a naive start_date/end_date
> > https://github.com/apache/incubator-airflow/pull/3822
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2979 Deprecated Celery
> > Option not in Options list
> > https://github.com/apache/incubator-airflow/pull/3832
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2989 No Parameter to
> change
> > bootDiskType for DataprocClusterCreateOperator
> > https://github.com/apache/incubator-airflow/pull/3825
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2990 Docstrings for
> > Hooks/Operators are in incorrect format
> > https://github.com/apache/incubator-airflow/pull/3820
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2994 flatten_results in
> > BigQueryOperator/BigQueryHook should default to None
> > https://github.com/apache/incubator-airflow/pull/3829
> >
> >
> > In addition to those PRs which are already marked with Fix Version of
> > 1.10.1 I think we should also pull in these:
> >
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2713 Rename async variable
> > for Python 3.7.0 compatibility
> > https://github.com/apache/incubator-airflow/pull/3561
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2895 Prevent scheduler
> from
> > spamming heartbeats/logs
> > https://github.com/apache/incubator-airflow/pull/3747
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2921 A trivial
> > incorrectness in CeleryExecutor()
> > https://github.com/apache/incubator-airflow/pull/3773
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2866 Missing CSRF Token
> > Error on Web RBAC UI Create/Update Operations
> > https://github.com/apache/incubator-airflow/pull/3804
> >
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2951
> > https://github.com/apache/incubator-airflow/pull/3798 Update dag_run
> > table end_date when state change
> > (though as written it has a few other deps to cherry pick in, so will see
> > about this one)
> >
> >
>