[GitHub] [airflow] gaoyibin0001 commented on issue #15286: use PythonVirtualenvOperator with a prebuilt env

[GitBox <gi...@apache.org>]

gaoyibin0001 commented on issue #15286:
URL: https://github.com/apache/airflow/issues/15286#issuecomment-1046497988


   > @ManikandanUV We are doing it the following way for now:
   > 
   > ```python
   > env = vars_dict.get("conda_env", None)
   > path_to_python = f"/home/username/.conda{'/envs/'+env if env is not None else ''}/bin/python"
   > 
   > parse_files = BashOperator(
   >             task_id='parse-files',
   >             bash_command=f"{path_to_python} {abs_path_code}/my_repo/parse.py {files_to_parse}",
   >             env={"PATH": os.environ["PATH"],
   >                  "DB_CONN": db_conn}
   >         )
   > ```
   > 
   > We have an environment variable containing the conda-env name which is used to get the full path to the Python executable. Then, using a BashOperator, we can use the same environment again for different Tasks.
   > 
   > Additionally, we run an update to the environment if requirements changed (note that we are using poetry as package manager):
   > 
   > ```python
   > update_repo = BashOperator(
   >     task_id=f"update-repo-{folder}",
   >     bash_command=f"cd {abs_path_code}/{folder}; "
   >            "git checkout master; git stash; git stash drop; git pull"
   >     )
   > install_dependencies = BashOperator(
   >     task_id=f"install-dependencies-{folder}",
   >           bash_command=f"cd {abs_path_code}/{folder}; conda activate {env_name}; poetry install "
   >     )
   > update_repo >> install_dependencies
   > `
   > ``
   > ```
   
   may use "conda run -n env_name python xxx.py


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org