[jira] [Created] (ARROW-14967) [CI][Python] Ability to include pip packages in the conda environments

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-14967:
---------------------------------------------

             Summary: [CI][Python] Ability to include pip packages in the conda environments
                 Key: ARROW-14967
                 URL: https://issues.apache.org/jira/browse/ARROW-14967
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Continuous Integration, Python
            Reporter: Joris Van den Bossche


For creating various conda environments, we currently have files like {{conda_env_cpp.txt}}, {{conda_env_sphinx.txt}}, {{conda_env_python.txt}}, etc 

Those can then be combined to create a specific conda environment with the subset of features you want, eg from the python docs:

{code}
    conda create -y -n pyarrow-dev -c conda-forge \
        --file arrow/ci/conda_env_unix.txt \
        --file arrow/ci/conda_env_cpp.txt \
        --file arrow/ci/conda_env_python.txt \
        --file arrow/ci/conda_env_gandiva.txt \
        compilers \
        python=3.9 \
        pandas
{code}

or installed as additional packages into an existing one (eg {{conda install --file arrow/ci/conda_env_python.txt}} in conda-python.dockerfile).

One disadvantage of this approach is that you cannot (as far as I am aware) not list pip packages in those .txt files.   
You can do that with environment.yml files, but those then don't really compose together as we do with the txt files, I think.

cc [~kszucs]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)