[jira] [Commented] (AIRFLOW-2871) Harden and improve Read the Docs build environment

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572828#comment-16572828 ] 

Ash Berlin-Taylor commented on AIRFLOW-2871:
--------------------------------------------

One other thing that would be nice to do is to have mutliple versions of Docs on RTD, and set the default to the latest release, rather than Master.

There has been a bit of confusion in the chat channel over things in the docs not "working" because of this.

> Harden and improve Read the Docs build environment
> --------------------------------------------------
>
>                 Key: AIRFLOW-2871
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2871
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: docs, Documentation
>            Reporter: Taylor Edmiston
>            Assignee: Taylor Edmiston
>            Priority: Major
>
> h2. Context
> In the process of resolving AIRFLOW-2857 (via [PR 3703|https://github.com/apache/incubator-airflow/pull/3703]), I noticed some oddities in our Read the Docs (RTD) build environment especially around cached dependencies.  This motivates hardening and showing some love to our RTD setup.
> h2. Problem
> I dug into the RTD build logs for a moment to find some closure on the mock dependency discussed in PR #3703 above. I think that our RTD environment possibly has been working by coincidence off of cached dependencies.
> {code:java}
> python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --ignore-installed --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip .[doc,docker,gcp_api,emr]{code}
> The directory referenced by that --cache-dir arg earlier in the log happens to have mock installed already.
> {code:java}
> python /home/docs/checkouts/readthedocs.org/user_builds/airflow/envs/latest/bin/pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/airflow/.cache/pip Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx<1.8 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6{code}
>  Here are some logs where you can see that (view raw):
>  # Latest successful (Aug. 7, 2018. 9:21 a.m.) - [7602630|https://readthedocs.org/projects/airflow/builds/7602630/]
>  # Last unsuccessful before (1) (Aug. 5, 2018. 1:24 p.m.) - [7593052|https://readthedocs.org/projects/airflow/builds/7593052/]
>  # Last successful before (2) (July 18, 2018. 3:23 a.m.) - [7503718|https://readthedocs.org/projects/airflow/builds/7503718/]
>  # First build (2016) - [4150778|https://readthedocs.org/projects/airflow/builds/4150778/]
> It appears that mock and others have potentially been cached since the first RTD build in 2016 (4).
> These versions like mock==1.0.1 do not appear to be coming from anywhere in our current config in incubator-airflow; I believe they are installed as [core dependencies of RTD itself|https://github.com/rtfd/readthedocs.org/blob/ca7afe6577672e129ccfe63abe33561dc32a6651/readthedocs/doc_builder/python_environments.py#L220-L235].
> Some but not all of these dependencies get upgraded to newer versions further down in the build.  In the case of mock, we were getting lucky that mock==1.0.1 was a dependency of RTD and our setup inherited that old version which allowed the docs build to succeed.  (There might be other cases of dependencies like this too.)
> h2. Solution
> My proposed enhancements to harden and improve our RTD setup are:
>  * Hardening
>  ** Set our RTD build to use a virtual environment if it's not already
>  ** Set our RTD build to ignore packages outside of its virtualenv like dependencies of RTD itself
>  ** Specify any dependencies broken by ^
>  ** Test wiping a version in the build environment (not sure if this clears cache dir)
>  *** [https://docs.readthedocs.io/en/latest/guides/wipe-environment.html#wiping-a-build-environment]
>  *** [https://docs.readthedocs.io/en/latest/builds.html#deleting-a-stale-or-broken-build-environment]
>  ** Make build.image, python.version, etc explicit in yaml config
>  *** [https://docs.readthedocs.io/en/latest/yaml-config.html]
>  ** Test upgrading our RTD environment from CPython 2.x to using CPython 3.x
>  * Misc
>  ** Improve RTD project page to have tags and description
>  ** Lint YAML file
> Note: I don't yet have maintainer access for airflow on RTD which I believe this would require.  I am happy to take this issue if I can get that.  I have experience as an admin of another project on RTD (simple-salesforce).
> ----
> /cc Everyone who commented in PR #3703 - [~kaxilnaik], [~ashb], [~TaoFeng] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)