[GitHub] [airflow] uranusjr commented on a change in pull request #17349: #16037 Add support for passing templated requirements.txt in PythonVirtualenvOperator

[GitBox <gi...@apache.org>]

uranusjr commented on a change in pull request #17349:
URL: https://github.com/apache/airflow/pull/17349#discussion_r680677265



##########
File path: airflow/operators/python.py
##########
@@ -333,20 +337,31 @@ def __init__(
             templates_exts=templates_exts,
             **kwargs,
         )
-        self.requirements = list(requirements or [])
+        if not isinstance(requirements, str):
+            self.requirements = list(requirements or [])
+        else:
+            self.requirements = requirements
         self.string_args = string_args or []
         self.python_version = python_version
         self.use_dill = use_dill
         self.system_site_packages = system_site_packages
-        if not self.system_site_packages and self.use_dill and 'dill' not in self.requirements:
-            self.requirements.append('dill')
         self.pickling_library = dill if self.use_dill else pickle
 
+    def pre_execute(self, context: Any):
+        if isinstance(self.requirements, list):
+            return
+
+        import pkg_resources
+        self.requirements = [str(req) for req in pkg_resources.parse_requirements(self.requirements)]

Review comment:
       The requirements file format can do more than listing requirements, and those cannot be parsed by `parse_requirements` (I don’t remember whether setuptools would throw or just silently drop them.) So instead of trying to pre-parse the content, I would try to parse it to pip instead.
   
   * Remove `pre_execute`
   * Change `prepare_virtualenv` to accept `requirements` as str (plus the currently supported list form)
   * When doing `pip install`, write the str type `requirements` to a temporary file for `pip intall -r`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org