[jira] [Updated] (AIRFLOW-2871) Harden and improve Read the Docs build environment

["Kaxil Naik (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaxil Naik updated AIRFLOW-2871:
--------------------------------
    Attachment: screenshot-1.png

> Harden and improve Read the Docs build environment
> --------------------------------------------------
>
>                 Key: AIRFLOW-2871
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: docs, Documentation
>            Reporter: Taylor Edmiston
>            Assignee: Taylor Edmiston
>            Priority: Major
>         Attachments: screenshot-1.png, screenshot-2.png
>
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some oddities in our Read the Docs (RTD) build environment especially around cached dependencies.  This motivates hardening and showing some love to our RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock dependency discussed in PR #3703 above. I think that our RTD environment possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --ignore-installed --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens to have mock installed already.
> {code:java}
> python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in our current config in incubator-airflow; I believe they are installed as [core dependencies of RTD itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further down in the build.  In the case of mock, we were getting lucky that mock==1.0.1 was a dependency of RTD and our setup inherited that old version which allowed the docs build to succeed.  (There might be other cases of dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears cache dir)
>  *** [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe this would require.  I am happy to take this issue if I can get that.  I have experience as an admin of another project on RTD (simple-salesforce).
> ----
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)