Publishing alpha releases to PyPI?

[Ash Berlin-Taylor <as...@firemirror.com>]

(Bit of a long mail, skip to the end for specific ask, and why I think it is safe)

Yesterday we tried upgrading a test cluster of ours to the alpha of 1.9.0 but ran into a few snags around installation.

We don't actually install Airflow directly; instead we install our in-house python module which lists apache-airflow as one of its requirements in setup.py:

setup(
    version=version,
    license='private-non-opensource',
    # ....
    install_requires=[
        'apache-airflow[s3, emr]>=1.8.1',
        'boto~=2.45',
        'smart_open~=1.3.5',
        'requests>=2.18.4',
        'requests_toolbelt>=0.8.0',
    ],
)

The problem is that although Pip can install from various sources (arbitrary https locations, direct from git tags etc) this doesn't work when pip is fetching requirements for another module. Meaning right now we can't just update the version in our setup.py and have it work, we have to install Airflow beforehand.

Now, about alpha releases on PyPi. By default pip will not install pre-release versions unless asked to[1] - it can be asked by using `pip install --pre`, or by asking for a pre-release version such as `pip install "apache-airflow>=1.9.0a1.

An example using the current Django 2.0 which is in beta release to show this behaviour.

Trying to install without a pre-release version:

  pip install 'django>=2.0'
  Collecting django>=2.0
    Could not find a version that satisfies the requirement django>=2.0 (from versions: 1.1.3, ..., 2.0a1, 2.0b1)
  No matching distribution found for django>=2.0

Versionless install ignores alpha and beta and goes to last "full" release:

  pip install 'django'
  Collecting django
    Downloading Django-1.11.6-py2.py3-none-any.whl (6.9MB)

Including 'a' in the requirement enables pre-releases and we get the beta. (You could also do `pip install --pre django`):

  pip install 'django>=2.0a'
  Collecting django>=2.0a
    Downloading Django-2.0b1-py3-none-any.whl (7.0MB)






I think uploading an alpha release to pypi is safe, as it won't get installed unless someone asks for it, and it makes easier for those who do. So what I'm asking for:

- Take the existing binary release that we've already got here https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha1/, and upload it to PyPi.
- Update release process to publish all alpha/beta/rc to PyPi in the future.

I'd be happy to update https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow if we start doing this (but that doesn't currently mention how uploads to pypi happen at all...)


Thanks,
Ash

[1]: https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions

---