[jira] [Created] (AIRFLOW-2871) Harden and improve Read the Docs build environment

["Taylor Edmiston (JIRA)" <ji...@apache.org>]

Taylor Edmiston created AIRFLOW-2871:
----------------------------------------

             Summary: Harden and improve Read the Docs build environment
                 Key: AIRFLOW-2871
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
             Project: Apache Airflow
          Issue Type: Bug
          Components: docs, Documentation
            Reporter: Taylor Edmiston
            Assignee: Taylor Edmiston


h2. Context

In the process of resolving AIRFLOW-2857 (via [PR 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some oddities in our Read the Docs (RTD) build environment especially around cached dependencies.  This motivates hardening and showing some love to our RTD setup.
h2. Problem

I dug into the RTD build logs for a moment to find some closure on the mock dependency discussed in PR #3703 above. I think that our RTD environment possibly has been working by coincidence off of cached dependencies.
{code:java}
python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --ignore-installed --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip .[doc,docker,gcp_api,emr]{code}
The directory referenced by that --cache-dir arg earlier in the log happens to have mock installed already.
{code:java}
python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
 Here are some logs where you can see that (view raw):
 # Latest successful (Aug. 7, 2018. 9:21 a.m.) - [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
 # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
 # Last successful before (2) (July 18, 2018. 3:23 a.m.) - [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
 # First build (2016) - [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]

It appears that mock and others have potentially been cached since the first RTD build in 2016 (4).

These versions like mock==1.0.1 do not appear to be coming from anywhere in our current config in incubator-airflow; I believe they are installed as [core dependencies of RTD itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].

Some but not all of these dependencies get upgraded to newer versions further down in the build.  In the case of mock, we were getting lucky that mock==1.0.1 was a dependency of RTD and our setup inherited that old version which allowed the docs build to succeed.  (There might be other cases of dependencies like this too.)
h2. Solution

My proposed enhancements to harden and improve our RTD setup are:
 * Hardening
 ** Set our RTD build to use a virtual environment if it's not already
 ** Set our RTD build to ignore packages outside of its virtualenv like dependencies of RTD itself
 ** Specify any dependencies broken by ^
 ** Test wiping a version in the build environment (not sure if this clears cache dir)
 *** [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
 *** [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
 ** Make build.image, python.version, etc explicit in yaml config
 *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
 ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
 * Misc
 ** Improve RTD project page to have tags and description
 ** Lint YAML file

Note: I don't yet have maintainer access for airflow on RTD which I believe this would require.  I am happy to take this issue if I can get that.  I have experience as an admin of another project on RTD (simple-salesforce).
----
/cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)