Using EMR hooks and operators (and Airflow release schedule)

[Ben Hoyt <be...@gmail.com>]

I'd like to start using the EMR hooks and operators (for example
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/emr_hook.py).
However, they were added on June 30, and the latest release is 1.7.1.3
dated June 13. So a couple of questions:

1) What is the best practice if I want newer features? Is running off the
latest master a terrible idea? How do other folks do this?
2) What's the release schedule? From the commit history it looks like a lot
of stuff has been added in the last few months but isn't available in a
release yet.

Somewhat relatedly, I didn't see any documentation on upgrading. I presume
there may be database migrations involved? How does one upgrade the db from
version X to version Y, or know if migrations are needed?

Thanks,
Ben

Re: Using EMR hooks and operators (and Airflow release schedule)

[Dennis O'Brien <de...@dennisobrien.net>]

Hi Ben,

I've run into a similar issue and so far have a sub-optimal solution.  The
main issue is that master is unstable, at least for my setup.  I filed an
issue here:  https://issues.apache.org/jira/browse/AIRFLOW-664  So in the
meantime, I've made a branch from 1.7.1.3 and apply my changes there, pip
installing from the github commit.  I'm also making pull requests, but
since I can't run master, I'm not able to test or feel confident I'm not
introducing a regression.  This problem with master might be with my
particular configuration -- at least I haven't heard anyone else having
this problem, and I've asked on gitter, the dev mailing list, and filed a
Jira issue.

For upgrading, I have a deploy process that calls `airflow initdb` on each
deploy.  `initdb` calls `upgradedb` so In the case of bumping the version,
it handles the schema upgrade.  I'm not sure if there is a graceful way to
handle downgrading, but on the few times I've had to do that, I've either
restored MySQL from a snapshot or just `airflow resetdb` depending on the
situation.

cheers,
Dennis



On Thu, Dec 1, 2016 at 10:54 AM Ben Hoyt <be...@gmail.com> wrote:

> I'd like to start using the EMR hooks and operators (for example
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/emr_hook.py
> ).
> However, they were added on June 30, and the latest release is 1.7.1.3
> dated June 13. So a couple of questions:
>
> 1) What is the best practice if I want newer features? Is running off the
> latest master a terrible idea? How do other folks do this?
> 2) What's the release schedule? From the commit history it looks like a lot
> of stuff has been added in the last few months but isn't available in a
> release yet.
>
> Somewhat relatedly, I didn't see any documentation on upgrading. I presume
> there may be database migrations involved? How does one upgrade the db from
> version X to version Y, or know if migrations are needed?
>
> Thanks,
> Ben
>