[jira] [Created] (BEAM-6542) Python ValidatesContainer test suite should verify that installed dependencies match dependencies in requirements file.

["Valentyn Tymofieiev (JIRA)" <ji...@apache.org>]

Valentyn Tymofieiev created BEAM-6542:
-----------------------------------------

             Summary: Python ValidatesContainer test suite should verify that installed dependencies match dependencies in requirements file.
                 Key: BEAM-6542
                 URL: https://issues.apache.org/jira/browse/BEAM-6542
             Project: Beam
          Issue Type: Improvement
          Components: sdk-py-core
            Reporter: Valentyn Tymofieiev
            Assignee: Ahmet Altay


Python [ValidatesContainer test suites|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy] build Docker containers and run some integration tests using those containers, to make sure that containers can be built and used. A python container that we build [includes Beam SDK|https://github.com/apache/beam/blob/1a6490d3fd9245fc59838bd4bd531755304a855a/sdks/python/container/Dockerfile#L47].

During container build we install several pip packages, which is influenced by [requirements.txt|https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt], [SDK dependencies|https://github.com/apache/beam/blob/d9a1bac19c52b92804a204b7ca881b3e8617b42c/sdks/python/setup.py#L193], and downstream dependencies of packages we install.

The purpose of specifying dependencies in setup.py is to define minimal viable requirements for SDK to be installed.

The purpose of requirements.txt is to configure the runtime environment for SDK harness as and requires more precision to make sure there are no version dependency conflicts and to make sure that versions of dependencies installed in the container match across two different container builds. therefore we should specify exact versions in requirements.txt and also include all transitive dependencies of Beam.

Unfortunately, requirements.txt can easily go out of sync even with Beam SDK requirements ([example|https://github.com/apache/beam/pull/7657])

We should strengthen ValidatesContainer test suite to verify that version of dependencies installed in the container matches versions specified in requirements.txt.

One possible way to do it is to run `pip freeze` and compare the output with requirements.txt.

cc [~markflyhigh], [~altay].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)