[pulsar] branch master updated: improve python function packaging (#2965)

[si...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new c704aaa  improve python function packaging (#2965)
c704aaa is described below

commit c704aaa727bee1d91019e50e841cc30977f17ebc
Author: Boyang Jerry Peng <je...@gmail.com>
AuthorDate: Fri Nov 9 17:26:32 2018 -0800

    improve python function packaging (#2965)
    
    ### Motivation
    
    Current python function package of a zip file requires a requirements.txt to be present. It also requires all your dependencies to be explicitly mentioned in requirements.txt.  This is cumbersome and unwieldy especially if the user built a wheel file locally and want to include it as a dependency.
    
    Going forward, python instance should just install all wheel files in the deps folder and not rely on requirements.txt for a list of dependencies.  Users can just drop their wheel files in the deps folder and then their are good to go.  I think this is what AWS lambda also does for their python packaging
---
 .../instance/src/main/python/python_instance_main.py | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/pulsar-functions/instance/src/main/python/python_instance_main.py b/pulsar-functions/instance/src/main/python/python_instance_main.py
index 2a02380..c6bf885 100644
--- a/pulsar-functions/instance/src/main/python/python_instance_main.py
+++ b/pulsar-functions/instance/src/main/python/python_instance_main.py
@@ -120,13 +120,19 @@ def main():
     zpfile = zipfile.ZipFile(str(args.py), 'r')
     zpfile.extractall(os.path.dirname(str(args.py)))
     basename = os.path.splitext(str(args.py))[0]
-    requirements_txt_file = os.path.join(os.path.dirname(str(args.py)), basename, "requirements.txt")
-    deps_file = os.path.join(os.path.dirname(str(args.py)), basename, "deps")
-    cmd = "pip install -t %s -r %s --no-index --find-links %s" % (os.path.dirname(str(args.py)), requirements_txt_file, deps_file)
-    retval = os.system(cmd)
-    if retval != 0:
-      print("Could not install user depedencies specified by the zip file")
-      sys.exit(1)
+
+    deps_dir = os.path.join(os.path.dirname(str(args.py)), basename, "deps")
+
+    if os.path.isdir(deps_dir) and os.listdir(deps_dir):
+      # get all wheel files from deps directory
+      wheel_file_list = [os.path.join(deps_dir, f) for f in os.listdir(deps_dir) if os.path.isfile(os.path.join(deps_dir, f)) and os.path.splitext(f)[1] =='.whl']
+      cmd = "pip install -t %s --no-index --find-links %s %s" % (os.path.dirname(str(args.py)), deps_dir, " ".join(wheel_file_list))
+      Log.debug("Install python dependencies via cmd: %s" % cmd)
+      retval = os.system(cmd)
+      if retval != 0:
+        print("Could not install user depedencies specified by the zip file")
+        sys.exit(1)
+    # add python user src directory to path
     sys.path.insert(0, os.path.join(os.path.dirname(str(args.py)), basename, "src"))
 
   log_file = os.path.join(args.logging_directory,