You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by cr...@gmail.com,
cr...@gmail.com on 2018/05/19 12:15:48 UTC
Bugs in apache-airflow 1.9.0, what should I upgrade to for the fixes?
Hi,
I am working on upgrading an installation of airflow from 1.7.0 to 1.9.0.
I have a test system, and things seem to be mostly working fine, but I encountered two annoying
problems:
https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1885
-> Fixed by: https://github.com/apache/incubator-airflow/pull/2844
https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1908
-> Fixed by: https://github.com/apache/incubator-airflow/pull/2867
These fixes are not in any released version (yet), and they are not in the v1-9-stable branch.
Would it be possible to cherry-pick these fixes into v1-9-stable? I am OK if another 1.9.x release is never done.
I am trying to upgrade to a newer, yet stable version of airflow. I looked at v1-10-test, but
there are a lot of changes from 1.9.0, and some database schema changes as well, so this
makes my upgrade harder, since I am trying to go with the 1.9.0 database schema.
Also, v1-10-test and master branches still seem to have a lot of changes going into it, so it is
hard for me to keep up.
If PR 2844 and PR 2867 were merged to v1-9-stable, that would make it easier for
me to pip install from a git branch.
Then later on, when v1.10 is released, I can upgrade to that.
Please help me, and advise on the best path forward.
Thanks.
--
Craig
--
Craig
--
Craig
Re: Bugs in apache-airflow 1.9.0, what should I upgrade to for the fixes?
Posted by cr...@gmail.com,
cr...@gmail.com.
On 2018/05/20 09:50:34, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:
> Hi Craig,
>
> Thanks for reaching out. My advise would be to work on a branch of Airflow
> which you can easily install with Pip:
> pip install git+https://github.comapache/incubator-airflow
> @airflow-1-9-patched
Fokko,
Thanks for your response.
Based on your response, I want to try to stick to using the Airflow github repository
from a specific git revision if possible.
I created a requirements.txt with two lines:
git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
kombu==4.1.0
I then did:
1. create a virtual environment
2. pip install -r requirements.txt
3. airflow webserver
When I look at the version in the web interface, it shows a version of:
2.0.0.dev0+incubating
even though I used the v1-10-test branch.
It looks like these two commits got merged to v1-10-test branch which bump the version to 2.0:
apache/incubator-airflow@305a787
apache/incubator-airflow@a30acaf
That seems wrong for v1-10-test branch.
Do you know what the timeline for airflow 1.10 is?
I think I want to get my stuff to work with airflow v1-10-test branch this week,
using a requirements.txt like what I mentioned above.
Then when airflow 1.10 is officially released, I can make a quick change to pin the requirements.txt
at airflow 1.10.
--
Craig
Re: Bugs in apache-airflow 1.9.0, what should I upgrade to for the fixes?
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Craig,
Thanks for reaching out. My advise would be to work on a branch of Airflow
which you can easily install with Pip:
pip install git+https://github.comapache/incubator-airflow
@airflow-1-9-patched
The changes that you propose in the PR is mostly changes in the config,
which you should copy to your own config. In Airflow we've chosen the patch
of throwing an Exception when required parameters are not set. Which makes
it impossible to miss certain configuration settings. My suggestion would
be:
- Make the config up to date by comparing your exsisting config with the
one on the master branch.
- Update the fields in your database to upgrade to the latest schema.
Version 1.10 might take some work because a lot of columns are converted to
be datetime aware.
Hope this helps, if you have any further questions, please don't hestitate
to drop a line on the dev mailinglist.
Cheers, Fokko
2018-05-19 17:26 GMT+02:00 Taylor Edmiston <te...@gmail.com>:
> I know it's not an ideal longterm solution, but for similar use cases in
> the past, I've just cherry picked the commits of interest onto the my own
> (soft) fork of Airflow.
>
> Taylor
>
> Taylor Edmiston
> Blog <https://blog.tedmiston.com/> | CV <https://stackoverflow.com/cv/
> taylor> | LinkedIn <https://www.linkedin.com/in/tedmiston/> | AngelList <
> https://angel.co/taylor> | Stack Overflow <https://stackoverflow.com/
> users/149428/taylor-edmiston>
>
>
> > On May 19, 2018, at 8:15 AM, crodr001@gmail.com wrote:
> >
> > Hi,
> >
> > I am working on upgrading an installation of airflow from 1.7.0 to 1.9.0.
> >
> > I have a test system, and things seem to be mostly working fine, but I
> encountered two annoying
> > problems:
> >
> > https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1885
> > -> Fixed by: https://github.com/apache/incubator-airflow/pull/2844
> >
> > https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1908
> > -> Fixed by: https://github.com/apache/incubator-airflow/pull/2867
> >
> > These fixes are not in any released version (yet), and they are not in
> the v1-9-stable branch.
> >
> > Would it be possible to cherry-pick these fixes into v1-9-stable? I am
> OK if another 1.9.x release is never done.
> >
> > I am trying to upgrade to a newer, yet stable version of airflow. I
> looked at v1-10-test, but
> > there are a lot of changes from 1.9.0, and some database schema changes
> as well, so this
> > makes my upgrade harder, since I am trying to go with the 1.9.0 database
> schema.
> >
> > Also, v1-10-test and master branches still seem to have a lot of changes
> going into it, so it is
> > hard for me to keep up.
> >
> > If PR 2844 and PR 2867 were merged to v1-9-stable, that would make it
> easier for
> > me to pip install from a git branch.
> > Then later on, when v1.10 is released, I can upgrade to that.
> >
> > Please help me, and advise on the best path forward.
> >
> > Thanks.
> >
> > --
> > Craig
> >
> > --
> > Craig
> >
> > --
> > Craig
> >
>
>
Re: Bugs in apache-airflow 1.9.0, what should I upgrade to for the
fixes?
Posted by Taylor Edmiston <te...@gmail.com>.
I know it's not an ideal longterm solution, but for similar use cases in the past, I've just cherry picked the commits of interest onto the my own (soft) fork of Airflow.
Taylor
Taylor Edmiston
Blog <https://blog.tedmiston.com/> | CV <https://stackoverflow.com/cv/taylor> | LinkedIn <https://www.linkedin.com/in/tedmiston/> | AngelList <https://angel.co/taylor> | Stack Overflow <https://stackoverflow.com/users/149428/taylor-edmiston>
> On May 19, 2018, at 8:15 AM, crodr001@gmail.com wrote:
>
> Hi,
>
> I am working on upgrading an installation of airflow from 1.7.0 to 1.9.0.
>
> I have a test system, and things seem to be mostly working fine, but I encountered two annoying
> problems:
>
> https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1885
> -> Fixed by: https://github.com/apache/incubator-airflow/pull/2844
>
> https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1908
> -> Fixed by: https://github.com/apache/incubator-airflow/pull/2867
>
> These fixes are not in any released version (yet), and they are not in the v1-9-stable branch.
>
> Would it be possible to cherry-pick these fixes into v1-9-stable? I am OK if another 1.9.x release is never done.
>
> I am trying to upgrade to a newer, yet stable version of airflow. I looked at v1-10-test, but
> there are a lot of changes from 1.9.0, and some database schema changes as well, so this
> makes my upgrade harder, since I am trying to go with the 1.9.0 database schema.
>
> Also, v1-10-test and master branches still seem to have a lot of changes going into it, so it is
> hard for me to keep up.
>
> If PR 2844 and PR 2867 were merged to v1-9-stable, that would make it easier for
> me to pip install from a git branch.
> Then later on, when v1.10 is released, I can upgrade to that.
>
> Please help me, and advise on the best path forward.
>
> Thanks.
>
> --
> Craig
>
> --
> Craig
>
> --
> Craig
>