[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41946: [SPARK-44264][PYTHON][ML] FunctionPickler Class

["HyukjinKwon (via GitHub)" <gi...@apache.org>]

HyukjinKwon commented on code in PR #41946:
URL: https://github.com/apache/spark/pull/41946#discussion_r1263146395


##########
python/pyspark/ml/dl_util.py:
##########
@@ -0,0 +1,142 @@
+import os
+import tempfile
+import textwrap
+from typing import Any, Callable
+
+from pyspark import cloudpickle
+
+
+class FunctionPickler:
+    """
+    This class provides a way to pickle a function and its arguments.
+    It also provides a way to create a script that can run a
+    function with arguments if they have them pickled to a file.
+    It also provides a way of extracting the conents of a pickle file.
+    """
+
+    @staticmethod
+    def pickle_fn_and_save(
+        fn: Callable, file_path: str, save_dir: str, *args: Any, **kwargs: Any
+    ) -> str:
+        """
+        Given a function and args, this function will pickle them to a file.
+
+        Parameters
+        ----------
+        fn: Callable
+            The picklable function that will be pickled to a file.
+

Review Comment:
   I think we should remove a space between each prameters per https://numpydoc.readthedocs.io/en/latest/format.html#parameters



##########
python/pyspark/ml/dl_util.py:
##########
@@ -0,0 +1,142 @@
+import os
+import tempfile
+import textwrap
+from typing import Any, Callable
+
+from pyspark import cloudpickle
+
+
+class FunctionPickler:
+    """
+    This class provides a way to pickle a function and its arguments.
+    It also provides a way to create a script that can run a
+    function with arguments if they have them pickled to a file.
+    It also provides a way of extracting the conents of a pickle file.
+    """
+
+    @staticmethod
+    def pickle_fn_and_save(
+        fn: Callable, file_path: str, save_dir: str, *args: Any, **kwargs: Any
+    ) -> str:
+        """
+        Given a function and args, this function will pickle them to a file.
+
+        Parameters
+        ----------
+        fn: Callable
+            The picklable function that will be pickled to a file.
+
+        file_path: str
+            The path where to save the pickled function, args, and kwargs. If it's the
+            empty string, the function will decide on a random name.
+
+        save_dir: str
+            The directory in which to save the file with the pickled function and arguments.
+            Does nothing if the path is specified. If both file_path and save_dir are empty,
+            the function will write the file to the current working directory with a random
+            name.
+
+        *args: Any

Review Comment:
   ```suggestion
           *args
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org