You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2018/11/28 22:40:55 UTC
Call for fixes for Airflow 1.10.2
Hi everyone,
I'm starting the process of gathering fixes for a 1.10.2. So far the list
of issues I have that we should pull in are
*https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
<https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2>*
I will start pushing these as cherry-picked commits to the v1-10-test
branch today.
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by airflowuser <ai...@protonmail.com.INVALID>.
Some more PRs to consider for 1.10.2 :
https://issues.apache.org/jira/browse/AIRFLOW-3378 KubernetesPodOperator does not delete on timeout failure
https://issues.apache.org/jira/browse/AIRFLOW-3375 Support returning multiple tasks with BranchPythonOperator
https://issues.apache.org/jira/browse/AIRFLOW-3371 BigQueryHook's Ability to Create View
https://issues.apache.org/jira/browse/AIRFLOW-3332 Add BigQuery Streaming insert_all to BigQueryHook
https://issues.apache.org/jira/browse/AIRFLOW-3205 GCS: Support multipart upload
https://issues.apache.org/jira/browse/AIRFLOW-2814 Default Arg "file_process_interval" for class SchedulerJob is inconsistent with doc
https://issues.apache.org/jira/browse/AIRFLOW-2807 Add support for External ID when using STS Assume Role
https://issues.apache.org/jira/browse/AIRFLOW-1262 Allow configuration of email alert subject and body
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, December 12, 2018 7:27 PM, Daniel Imberman <da...@gmail.com> wrote:
> I'd also say that this https://github.com/apache/incubator-airflow/pull/3683 PR
> which allows users to bake their DAGs into their docker image is pretty
> critical (didn't realize this hadn't been merged yet). It's also ready to
> go if someone could PTAL :)
>
> On Thu, Dec 6, 2018 at 7:13 AM Daniel Imberman daniel.imberman@gmail.com
> wrote:
>
> > Thank you for setting this up Kaxil!
> > I know that
> > https://issues.apache.org/jira/plugins/servlet/mobile#issue/AIRFLOW-3281
> > is pretty necessary as without it a major portion of the git-sync for the
> > k8s executor is broken. Hopefully should be merged soon
> > On Thu, Dec 6, 2018, 6:13 AM James Meickle jmeickle@quantopian.com.invalid
> > wrote:
> >
> > > I suggest at least adding a commit to remove the broken S3 logging section
> > > I just reported here: https://issues.apache.org/jira/browse/AIRFLOW-3449
> > > On Wed, Nov 28, 2018 at 5:41 PM Kaxil Naik kaxilnaik@gmail.com wrote:
> > >
> > > > Hi everyone,
> > > > I'm starting the process of gathering fixes for a 1.10.2. So far the
> > > > list
> > > > of issues I have that we should pull in are
> > > > *
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project %3D AIRFLOW AND status %3D Resolved AND fixVersion %3D 1.10.2
> > >
> > > > <
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project %3D AIRFLOW AND status %3D Resolved AND fixVersion %3D 1.10.2
> > >
> > > > > -
> > > >
> > > > I will start pushing these as cherry-picked commits to the v1-10-test
> > > > branch today.
> > > > Kaxil Naik
> > > > *Big Data Consultant *@ Data Reply UK
> > > > *Certified *Google Cloud Data Engineer | Certified Apache Spark &
> > > > Neo4j
> > > > Developer
> > > > *Phone: *+44 (0) 74820 88992
> > > > LinkedIn: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by Daniel Imberman <da...@gmail.com>.
I'd also say that this https://github.com/apache/incubator-airflow/pull/3683 PR
which allows users to bake their DAGs into their docker image is pretty
critical (didn't realize this hadn't been merged yet). It's also ready to
go if someone could PTAL :)
On Thu, Dec 6, 2018 at 7:13 AM Daniel Imberman <da...@gmail.com>
wrote:
> Thank you for setting this up Kaxil!
>
> I know that
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/AIRFLOW-3281
> is pretty necessary as without it a major portion of the git-sync for the
> k8s executor is broken. Hopefully should be merged soon
>
>
> On Thu, Dec 6, 2018, 6:13 AM James Meickle <jm...@quantopian.com.invalid>
> wrote:
>
>> I suggest at least adding a commit to remove the broken S3 logging section
>> I just reported here: https://issues.apache.org/jira/browse/AIRFLOW-3449
>>
>> On Wed, Nov 28, 2018 at 5:41 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> > Hi everyone,
>> >
>> > I'm starting the process of gathering fixes for a 1.10.2. So far the
>> list
>> > of issues I have that we should pull in are
>> > *
>> >
>> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
>> > <
>> >
>> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
>> > >*
>> >
>> > I will start pushing these as cherry-picked commits to the v1-10-test
>> > branch today.
>> >
>> > *Kaxil Naik*
>> > *Big Data Consultant *@ *Data Reply UK*
>> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>> Neo4j
>> > Developer
>> > *Phone: *+44 (0) 74820 88992
>> > *LinkedIn*: https://www.linkedin.com/in/kaxil
>> >
>>
>
Re: Call for fixes for Airflow 1.10.2
Posted by Daniel Imberman <da...@gmail.com>.
Thank you for setting this up Kaxil!
I know that
https://issues.apache.org/jira/plugins/servlet/mobile#issue/AIRFLOW-3281 is
pretty necessary as without it a major portion of the git-sync for the k8s
executor is broken. Hopefully should be merged soon
On Thu, Dec 6, 2018, 6:13 AM James Meickle <jm...@quantopian.com.invalid>
wrote:
> I suggest at least adding a commit to remove the broken S3 logging section
> I just reported here: https://issues.apache.org/jira/browse/AIRFLOW-3449
>
> On Wed, Nov 28, 2018 at 5:41 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I'm starting the process of gathering fixes for a 1.10.2. So far the list
> > of issues I have that we should pull in are
> > *
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > <
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > >*
> >
> > I will start pushing these as cherry-picked commits to the v1-10-test
> > branch today.
> >
> > *Kaxil Naik*
> > *Big Data Consultant *@ *Data Reply UK*
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> > Developer
> > *Phone: *+44 (0) 74820 88992
> > *LinkedIn*: https://www.linkedin.com/in/kaxil
> >
>
Re: Call for fixes for Airflow 1.10.2
Posted by James Meickle <jm...@quantopian.com.INVALID>.
I suggest at least adding a commit to remove the broken S3 logging section
I just reported here: https://issues.apache.org/jira/browse/AIRFLOW-3449
On Wed, Nov 28, 2018 at 5:41 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi everyone,
>
> I'm starting the process of gathering fixes for a 1.10.2. So far the list
> of issues I have that we should pull in are
> *
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> <
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >*
>
> I will start pushing these as cherry-picked commits to the v1-10-test
> branch today.
>
> *Kaxil Naik*
> *Big Data Consultant *@ *Data Reply UK*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *Phone: *+44 (0) 74820 88992
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>
Re: Call for fixes for Airflow 1.10.2
Posted by airflowuser <ai...@protonmail.com.INVALID>.
Possible tickets to consider (All Prs merged and point to 2.0.0):
https://issues.apache.org/jira/browse/AIRFLOW-3378
https://issues.apache.org/jira/browse/AIRFLOW-3375
https://issues.apache.org/jira/browse/AIRFLOW-3371
https://issues.apache.org/jira/browse/AIRFLOW-3332
https://issues.apache.org/jira/browse/AIRFLOW-3205
https://issues.apache.org/jira/browse/AIRFLOW-2529
https://issues.apache.org/jira/browse/AIRFLOW-2814
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, December 1, 2018 11:36 PM, Ash Berlin-Taylor <as...@apache.org> wrote:
> I'd like to get https://issues.apache.org/jira/browse/AIRFLOW-3422 in if we can - but we need a fix first (Bolke: it's our favourite! DST time zones in next_schedule!) I'll take a look at this ... soon.
>
> -ash
>
> > On 28 Nov 2018, at 22:40, Kaxil Naik kaxilnaik@gmail.com wrote:
> > Hi everyone,
> > I'm starting the process of gathering fixes for a 1.10.2. So far the list
> > of issues I have that we should pull in are
> > https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project %3D AIRFLOW AND status %3D Resolved AND fixVersion %3D 1.10.2
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project %3D AIRFLOW AND status %3D Resolved AND fixVersion %3D 1.10.2I will start pushing these as cherry-picked commits to the v1-10-test
> > branch today.
> > Kaxil Naik
> > *Big Data Consultant *@ Data Reply UK
> > *Certified *Google Cloud Data Engineer | Certified Apache Spark & Neo4j
> > Developer
> > *Phone: *+44 (0) 74820 88992
> > LinkedIn: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by Ash Berlin-Taylor <as...@apache.org>.
I'd like to get https://issues.apache.org/jira/browse/AIRFLOW-3422 in if we can - but we need a fix first (Bolke: it's our favourite! DST time zones in next_schedule!) I'll take a look at this ... soon.
-ash
> On 28 Nov 2018, at 22:40, Kaxil Naik <ka...@gmail.com> wrote:
>
> Hi everyone,
>
> I'm starting the process of gathering fixes for a 1.10.2. So far the list
> of issues I have that we should pull in are
> *https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> <https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2>*
>
> I will start pushing these as cherry-picked commits to the v1-10-test
> branch today.
>
> *Kaxil Naik*
> *Big Data Consultant *@ *Data Reply UK*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *Phone: *+44 (0) 74820 88992
> *LinkedIn*: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by Kaxil Naik <ka...@gmail.com>.
Added a comment on the PR. Happy to merge it once that is addressed :)
On Thu, Nov 29, 2018 at 10:47 PM Kevin Yang <yr...@gmail.com> wrote:
> Hi Kaxil,
> Ty for running 1.10.2, that's pretty fast. Can you please also include the
> following bug fix in 1.10.2 please?
>
> AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
> reload_module in DagFileProcessorAgent
>
> Cheers,
> Kevin Y
>
> On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I'm starting the process of gathering fixes for a 1.10.2. So far the list
> > of issues I have that we should pull in are
> > *
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > <
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > >*
> >
> > I will start pushing these as cherry-picked commits to the v1-10-test
> > branch today.
> >
> > *Kaxil Naik*
> > *Big Data Consultant *@ *Data Reply UK*
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> > Developer
> > *Phone: *+44 (0) 74820 88992
> > *LinkedIn*: https://www.linkedin.com/in/kaxil
> >
>
--
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by Kaxil Naik <ka...@gmail.com>.
Added.
On Sat, Dec 1, 2018 at 2:25 AM Deng Xiaodong <xd...@gmail.com> wrote:
> Thanks Kaxil.
>
> Not sure if 1.10.2 is focusing on fixing 1.10.0/1.10.1, or also open for
> new patches/features?
>
> The current `airflow flower` (until 1.10.1) doesn’t come with any
> authentication. This may make essential information exposed to unintended
> users in an untrusted environment.
>
> Earlier I submitted a PR to add HTTP basic authentication for Flower in
> Airflow. This feature helps enhance the the whole application slightly in
> terms of security.
>
> Please refer to:
> PR 4166 (merged in master):
> https://github.com/apache/incubator-airflow/pull/4166 <
> https://github.com/apache/incubator-airflow/pull/4166>
> JIRA: https://issues.apache.org/jira/browse/AIRFLOW-3323 <
> https://issues.apache.org/jira/browse/AIRFLOW-3323>
>
> Thanks.
>
>
> XD
>
>
> > On 30 Nov 2018, at 6:47 AM, Kevin Yang <yr...@gmail.com> wrote:
> >
> > Hi Kaxil,
> > Ty for running 1.10.2, that's pretty fast. Can you please also include
> the
> > following bug fix in 1.10.2 please?
> >
> > AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
> > reload_module in DagFileProcessorAgent
> >
> > Cheers,
> > Kevin Y
> >
> > On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> >> Hi everyone,
> >>
> >> I'm starting the process of gathering fixes for a 1.10.2. So far the
> list
> >> of issues I have that we should pull in are
> >> *
> >>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >> <
> >>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >>> *
> >>
> >> I will start pushing these as cherry-picked commits to the v1-10-test
> >> branch today.
> >>
> >> *Kaxil Naik*
> >> *Big Data Consultant *@ *Data Reply UK*
> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> >> Developer
> >> *Phone: *+44 (0) 74820 88992
> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
>
>
--
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil
Re: Call for fixes for Airflow 1.10.2
Posted by Deng Xiaodong <xd...@gmail.com>.
Thanks Kaxil.
Not sure if 1.10.2 is focusing on fixing 1.10.0/1.10.1, or also open for new patches/features?
The current `airflow flower` (until 1.10.1) doesn’t come with any authentication. This may make essential information exposed to unintended users in an untrusted environment.
Earlier I submitted a PR to add HTTP basic authentication for Flower in Airflow. This feature helps enhance the the whole application slightly in terms of security.
Please refer to:
PR 4166 (merged in master): https://github.com/apache/incubator-airflow/pull/4166 <https://github.com/apache/incubator-airflow/pull/4166>
JIRA: https://issues.apache.org/jira/browse/AIRFLOW-3323 <https://issues.apache.org/jira/browse/AIRFLOW-3323>
Thanks.
XD
> On 30 Nov 2018, at 6:47 AM, Kevin Yang <yr...@gmail.com> wrote:
>
> Hi Kaxil,
> Ty for running 1.10.2, that's pretty fast. Can you please also include the
> following bug fix in 1.10.2 please?
>
> AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
> reload_module in DagFileProcessorAgent
>
> Cheers,
> Kevin Y
>
> On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I'm starting the process of gathering fixes for a 1.10.2. So far the list
>> of issues I have that we should pull in are
>> *
>> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
>> <
>> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
>>> *
>>
>> I will start pushing these as cherry-picked commits to the v1-10-test
>> branch today.
>>
>> *Kaxil Naik*
>> *Big Data Consultant *@ *Data Reply UK*
>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
>> Developer
>> *Phone: *+44 (0) 74820 88992
>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>
Re: Call for fixes for Airflow 1.10.2
Posted by Kevin Yang <yr...@gmail.com>.
Hi Kaxil,
Ty for running 1.10.2, that's pretty fast. Can you please also include the
following bug fix in 1.10.2 please?
AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
reload_module in DagFileProcessorAgent
Cheers,
Kevin Y
On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi everyone,
>
> I'm starting the process of gathering fixes for a 1.10.2. So far the list
> of issues I have that we should pull in are
> *
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> <
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >*
>
> I will start pushing these as cherry-picked commits to the v1-10-test
> branch today.
>
> *Kaxil Naik*
> *Big Data Consultant *@ *Data Reply UK*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *Phone: *+44 (0) 74820 88992
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>