[jira] [Updated] (BEAM-6542) Python ValidatesContainer test suite should verify that installed dependencies match dependencies in requirements file.

["Kenneth Knowles (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-6542:
----------------------------------
    Resolution: Fixed
        Status: Resolved  (was: Resolved)

Hello! Due to a bug in our Jira configuration, this issue had status:Resolved but resolution:Unresolved.

I am bulk editing these issues to have resolution:Fixed

If a different resolution is appropriate, please change it. To do this, click the "Resolve" button (you can do this even for closed issues) and set the Resolution field to the right value.

> Python ValidatesContainer test suite should verify that installed dependencies match dependencies in requirements file.
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-6542
>                 URL: https://issues.apache.org/jira/browse/BEAM-6542
>             Project: Beam
>          Issue Type: Improvement
>          Components: testing
>            Reporter: Valentyn Tymofieiev
>            Priority: P3
>
> Python [ValidatesContainer test suites|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy] build Docker containers and run some integration tests using those containers, to make sure that containers can be built and used. A python container that we build [includes Beam SDK|https://github.com/apache/beam/blob/1a6490d3fd9245fc59838bd4bd531755304a855a/sdks/python/container/Dockerfile#L47].
> During container build we install several pip packages, which is influenced by [requirements.txt|https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt], [SDK dependencies|https://github.com/apache/beam/blob/d9a1bac19c52b92804a204b7ca881b3e8617b42c/sdks/python/setup.py#L193], and downstream dependencies of packages we install.
> The purpose of specifying dependencies in setup.py is to define minimal viable requirements for SDK to be installed.
> The purpose of requirements.txt is to configure the runtime environment for SDK harness as and requires more precision to make sure there are no version dependency conflicts and to make sure that versions of dependencies installed in the container match across two different container builds. therefore we should specify exact versions in requirements.txt and also include all transitive dependencies of Beam.
> Unfortunately, requirements.txt can easily go out of sync even with Beam SDK requirements ([example|https://github.com/apache/beam/pull/7657])
> We should strengthen ValidatesContainer test suite to verify that version of dependencies installed in the container matches versions specified in requirements.txt.
> One possible way to do it is to run `pip freeze` and compare the output with requirements.txt.
> cc [~markflyhigh], [~altay].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)