You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "kristynsmith (via GitHub)" <gi...@apache.org> on 2025/08/19 14:23:26 UTC

[PR] Integrate lambda name pickling with Cloudpickle [beam]

kristynsmith opened a new pull request, #35904:
URL: https://github.com/apache/beam/pull/35904

   A continuation of [https://github.com/apache/beam/pull/35656](https://www.google.com/url?sa=D&q=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fpull%2F35656) that uses the helper functions in code_object_pickler.py to patch pickler implementations used by Beam.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2415019370


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Oh I got confused. We need two functions, get_code_object_identifier and get_code_from_identifier
   
   We can add both of these to the config and use the functools.partial approach for get_code_from_identifier?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416960306


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   Is there something else we can test/assert here specific to the behavior of enable_stable_code_identifier_pickling? And in test_disable_stable_code_identifier_pickling we can assert the opposite



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417578769


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   Is there anything about the lambda that is different when pickled with this option? For example in this test I checked the co_filename of a type https://github.com/apache/beam/blob/ac4b5ab2aac87f1e1d71fc97637514b9b9292ef5/sdks/python/apache_beam/coders/coders_test_common.py#L838



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377243995


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   I am looking to see how to simplify your monkey patch, the below comment is concerning your original version excluded your commits from last iteration (https://github.com/apache/beam/compare/master...tvalentyn:subset1)
   
   > the first element that function_getnewargs returns is a string if the code object identifier was able to be created. in that case, we want to use make_function_from_identifier.
     
   Instead of modifying   `_function_getnewargs` could we instead have  the branching within `_dynamic_function_reduce`? It looks like `_function_getnewargs` is not used anywhere else. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2379276653


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   yes, i could do:
   ```
   (in _dynamic_function_reduce_)
   code_path = get_code_object_identifier(func) if self.config.enable_stable_code_identifier_pickling else None
   newargs = self._function_getnewargs(func)
   if code_path
     make_function = _make_function_from_identifier
     newargs = code_path
   else:
     make_function = _make_function
   return (make_function, newargs, {etc. etc.})
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2379312150


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock

Review Comment:
   are you saying in the context manager replace the `with.patch.object` part with  `cloudpickle._dynamic_function_reduce = _patched_dynamic_function_reduce`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3386270922

   my most recent commit isn't showing up but it's here: https://github.com/kristynsmith/pickle/commit/31dd574a652a3d40b53760bb6fad7d9200f8ab60


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3254279332

   Assigning reviewers:
   
   R: @liferoad for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2433380786


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +302,48 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_stable_identifier_uses_current_code(self):
+    pickler.set_library('cloudpickle')
+
+    # Get original dynamic function
+    func_v1 = module_test.mutable_test_function()
+
+    pickled_stable = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=True)
+
+    pickled_frozen = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=False)
+
+    # Save original function for cleanup
+    original_function = module_test.mutable_test_function
+
+    try:
+      # Monkey patch: Replace the entire outer function with v2
+      code_v2 = """
+def mutable_test_function():

Review Comment:
   Replace with
   ```
         code_v2 = textwrap.dedent("""
         def mutable_test_function():
           def dynamic_function():
             return "version2"
   
           return dynamic_function
         """)
   ```
   
   Also `import textwrap` into the pickler_test.py file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith closed pull request #35904: Integrate lambda name pickling with Cloudpickle
URL: https://github.com/apache/beam/pull/35904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416975820


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   no i'm still getting `TypeError: _make_function_from_identifier() missing 4 required positional arguments: 'globals', 'name', 'argdefs', and 'closure'`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377260012


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -78,6 +78,9 @@
 import warnings
 import weakref
 
+from apache_beam.internal.code_object_pickler import get_code_from_identifier

Review Comment:
   +1 to run the CP test, we can try that on your fork before merging. additng tests can be a separate PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2380092124


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -111,6 +111,7 @@ class CloudPickleConfig:
   """Configuration for cloudpickle behavior."""
   id_generator: typing.Optional[callable] = uuid_generator
   skip_reset_dynamic_type_state: bool = False
+  enable_stable_code_identifier_pickling: bool = False

Review Comment:
   please add description here, and rename the setting



##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock
 import zlib
 
 from apache_beam.internal.cloudpickle import cloudpickle
+from apache_beam.internal import code_object_pickler
 
 DEFAULT_CONFIG = cloudpickle.CloudPickleConfig(
-    skip_reset_dynamic_type_state=True)
+    skip_reset_dynamic_type_state=True,
+    enable_stable_function_identifiers=False)

Review Comment:
   please add description of the setting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2380090304


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock

Review Comment:
   import is no longer needed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2380106194


##########
sdks/python/apache_beam/runners/pipeline_context.py:
##########
@@ -227,6 +227,7 @@ def __init__(
     self.iterable_state_write = iterable_state_write
     self._requirements = set(requirements)
     self.enable_best_effort_deterministic_pickling = False
+    self.enable_lambda_name_pickling = False

Review Comment:
   i thought so, isn't this part of how to use the feature when setting it with a command line flag?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2437669722


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -127,10 +133,18 @@ class CloudPickleConfig:
             
         filepath_interceptor: Used to modify filepaths in `co_filename` and
             function.__globals__['__file__'].
+
+        get_code_object_params: Use identifiers derived from code

Review Comment:
   please update the docstring.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2421355459


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   I tried
   
   file: module_test.py
   ```
   def mutable_test_function():
       def dynamic_function():
           return "version1"
       return dynamic_function
   ```
   
   file: pickler_test.py
   ```
   def test_stable_identifier_uses_current_code(self):
       """Stable identifiers resolve to current code, frozen bytecode preserves original."""
       pickler.set_library('cloudpickle')
       import apache_beam.internal.module_test as module_test
       
       # Get original dynamic function
       func_v1 = module_test.mutable_test_function()
       
       pickled_stable = pickler.dumps(
           func_v1,
           enable_stable_code_identifier_pickling=True)
       
       pickled_frozen = pickler.dumps(
           func_v1,
           enable_stable_code_identifier_pickling=False)
       
       # Save original function for cleanup
       original_function = module_test.mutable_test_function
       
       try:
           # Monkey patch: Replace the entire outer function with v2
           code_v2 = """
   def mutable_test_function():
       def dynamic_function():
           return "version2"
       return dynamic_function
   """
           namespace = {}
           exec(code_v2, namespace)
           module_test.mutable_test_function = namespace['mutable_test_function']
           
           # Unpickle both
           func_stable = pickler.loads(pickled_stable)
           func_frozen = pickler.loads(pickled_frozen)
           
           # Stable identifier resolves to NEW code (version2)
           self.assertEqual("version2", func_stable())
           
           # Frozen bytecode uses OLD code (version1)
           self.assertEqual("version1", func_frozen())
           
       finally:
           # Restore original function
           module_test.mutable_test_function = original_function
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417052519


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1305,6 +1326,23 @@ class Pickler(pickle.Pickler):
 
   dispatch_table = ChainMap(_dispatch_table, copyreg.dispatch_table)
 
+  def _stable_identifier_function_reduce(self, func):
+    code_path = self.config.get_code_object_params.get_code_object_identifier(
+        func)
+    if not code_path:
+      return self._dynamic_function_reduce(func)
+    newargs = (code_path, )

Review Comment:
   See `newargs = self._function_getnewargs(func)` in `_dynamic_function_reduce`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2415005776


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Can you add another argument here which passes the function
   ```
   def _make_function_from_identifier(get_code_from_identifier, code_path, globals, name, argdefs, closure):
   ```
   
   Then in _stable_identifier_function_reduce you do
   ```
   return (
           functools.partial(_make_function_from_identifier, self.config.get_code_object_identifier), 
           newargs, 
           state,
           None,
           None,
           _function_setstate)
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2415040651


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Does this work https://github.com/apache/beam/compare/master...claudevdm:beam:cloudpickle-lambdas?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2433085413


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +302,48 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_stable_identifier_uses_current_code(self):
+    pickler.set_library('cloudpickle')
+
+    # Get original dynamic function
+    func_v1 = module_test.mutable_test_function()
+
+    pickled_stable = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=True)
+
+    pickled_frozen = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=False)
+
+    # Save original function for cleanup
+    original_function = module_test.mutable_test_function
+
+    try:
+      # Monkey patch: Replace the entire outer function with v2
+      code_v2 = """
+def mutable_test_function():

Review Comment:
   If this causes linting issues, you can indent and call textwrap.dedent like here https://github.com/apache/beam/blob/99ee1738e2bf33add4487de2dad4c290f847b4f1/sdks/python/apache_beam/coders/coders_test_common.py#L725



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417746957


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1305,6 +1325,30 @@ class Pickler(pickle.Pickler):
 
   dispatch_table = ChainMap(_dispatch_table, copyreg.dispatch_table)
 
+  def _stable_identifier_function_reduce(self, func):
+    code_object_params = self.config.get_code_object_params
+    if code_object_params is None:
+      return self._dynamic_function_reduce(func)
+    code_path = code_object_params.get_code_object_identifier(func)
+    if not code_path:
+      return self._dynamic_function_reduce(func)
+    newargs = (
+        code_path,
+        func.__globals__,

Review Comment:
   I think this is the issue the entire func.__globals__ is being pickled. We should filter the globals down like in https://github.com/apache/beam/blob/ac4b5ab2aac87f1e1d71fc97637514b9b9292ef5/sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py#L1347



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416978123


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)

Review Comment:
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417043894


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1305,6 +1326,23 @@ class Pickler(pickle.Pickler):
 
   dispatch_table = ChainMap(_dispatch_table, copyreg.dispatch_table)
 
+  def _stable_identifier_function_reduce(self, func):
+    code_path = self.config.get_code_object_params.get_code_object_identifier(
+        func)
+    if not code_path:
+      return self._dynamic_function_reduce(func)
+    newargs = (code_path, )

Review Comment:
   I think the issue here. newargs need to contain `newargs = (code_path, globals, name, argdefs, closure)`, because those are passed to _make_function_from_identifier



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3250963037

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2429375786


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,20 +302,48 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
-  def test_enable_stable_code_identifier_pickling(self):
-    pickler.set_library('cloudpickle')
-    pickled = pickler.loads(
-        pickler.dumps(lambda x: x, enable_stable_code_identifier_pickling=True))
-    pickled_type = type(pickled)
-    self.assertIsInstance(pickled, pickled_type)
-
-  def test_disable_stable_code_identifier_pickling(self):
-    pickler.set_library('cloudpickle')
-    pickled = pickler.loads(
-        pickler.dumps(
-            lambda x: x, enable_stable_code_identifier_pickling=False))
-    pickled_type = type(pickled)
-    self.assertIsInstance(pickled, pickled_type)
+def test_stable_identifier_uses_current_code(self):

Review Comment:
   It seems this test did not run. Can you fix indentation (needs two lines like other tests).
   
   Also define mutable_test_function in module_test.py
   ```
   def mutable_test_function():
       def dynamic_function():
           return "version1"
       return dynamic_function
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417614476


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   no the lambda stays the same
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2379276653


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   yes, i could do:
   ```
   code_path = get_code_object_identifier(func) if self.config.enable_stable_code_identifier_pickling else None
   newargs = self._function_getnewargs(func)
   if code_path
     make_function = _make_function_from_identifier
     newargs = code_path
   else:
     make_function = _make_function
   return (make_function, newargs, {etc. etc.})
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3426831801

   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417056301


##########
sdks/python/apache_beam/internal/pickler.py:
##########
@@ -47,8 +47,17 @@ def dumps(
     o,
     enable_trace=True,
     use_zlib=False,
-    enable_best_effort_determinism=False) -> bytes:
-
+    enable_best_effort_determinism=False,
+    enable_stable_code_identifier_pickling=False) -> bytes:
+
+  if (desired_pickle_lib == cloudpickle_pickler and
+      enable_stable_code_identifier_pickling):
+    return cloudpickle_pickler.dumps(
+        o,
+        enable_trace=enable_trace,
+        use_zlib=use_zlib,
+        enable_best_effort_determinism=enable_best_effort_determinism,
+        config=cloudpickle_pickler.STABLE_CODE_IDENTIFIER_PICKLING_CONFIG)

Review Comment:
   Passing a config is unnecessary with the changes you made in cloudpickle_pickler.py.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2415015553


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Also will code_path be deterministic on all environments?
   
   Cant the code_path just be passed to _make_function_from_identifier through functools.partial so get_code_from_identifier does not have to be called again?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2350284775


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -78,6 +78,9 @@
 import warnings
 import weakref
 
+from apache_beam.internal.code_object_pickler import get_code_from_identifier

Review Comment:
   Can we do the cloudpickle changes in a separate PR from the dill changes?
   
   I would also like us to run the official cloudpickle test suite modified here https://github.com/cloudpipe/cloudpickle/compare/master...claudevdm:cloudpickle:cloudpickle-configurable once this functionality is added to the CloudPickleConfig. 
   
   Maybe this is a good time to think about the right way to fork cloudpickle with test coverage @tvalentyn ?



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1326,7 +1343,12 @@ def dump(self, obj):
         raise
 
   def __init__(
-      self, file, protocol=None, buffer_callback=None, config=DEFAULT_CONFIG):
+      self,
+      file,
+      protocol=None,
+      buffer_callback=None,
+      config=DEFAULT_CONFIG,
+      enable_lambda_name=False):

Review Comment:
   +1, can we add this as an option to the CloudPickleConfig, and guard all the new logic in this PR with that config option  https://github.com/apache/beam/blob/bde162c5e20543d95ca44230ecda6bb973943150/sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py#L113?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2361147373


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   1. the first element that `function_getnewargs` returns is a string if the code object identifier was able to be created. in that case, we want to use make_function_from_identifier.
   2. not really sure what you mean here.



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1336,6 +1358,7 @@ def __init__(
     self.globals_ref = {}
     self.proto = int(protocol)
     self.config = config
+    self.enable_lambda_name = enable_lambda_name

Review Comment:
   my first instinct is `enable_code_identifier_pickling` since that would match the language used throughout this project. but it's not super clear to someone completely unfamiliar with this feature. in a docstring i might say something like "enables the use of a unique, stable code identifier for pickling" so maybe `enable_stable code_identifier_pickling` would be more clear.



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -78,6 +78,9 @@
 import warnings
 import weakref
 
+from apache_beam.internal.code_object_pickler import get_code_from_identifier

Review Comment:
   the only dill changes here are just for handling if the feature is enabled but that pickling library being used is dill. is that what you're referring to? this won't be integrated with dill.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3383029259

   > @kristynsmith can you please check failing tests
   
   i was able to fix the test but only by adding an import in cloudpickle.py. i know it's not ideal but i wasn't able to figure out how to fix it otherwise. in make_function_from_identifier, we were trying to have get_code_from_identifier be passed in as a parameter but  _stable_identifier_function_reduce has to return all of the parameters needed for make_function_from_identifier and there's no way to get get_code_from_identifier without importing it. so to simplify it i removed get_code_from_identifier as a parameter for make_function_from_identifier and had the reference to it in make_function_from_identifier come from the import. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2415005776


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Can you add another argument here which passes the function
   
   def _make_function_from_identifier(get_code_from_identifier, code_path, globals, name, argdefs, closure):
   
   Then in _stable_identifier_function_reduce you do
   ```
   return (
           functools.partial(_make_function_from_identifier), 
           newargs, self.config.get_code_object_identifier
           state,
           None,
           None,
           _function_setstate)
   ```
   
   



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -567,6 +577,11 @@ def _make_function(code, globals, name, argdefs, closure):
   return types.FunctionType(code, globals, name, argdefs, closure)
 
 
+def _make_function_from_identifier(code_path, globals, name, argdefs, closure):

Review Comment:
   Can you add another argument here which passes the function
   ```
   def _make_function_from_identifier(get_code_from_identifier, code_path, globals, name, argdefs, closure):
   ```
   
   Then in _stable_identifier_function_reduce you do
   ```
   return (
           functools.partial(_make_function_from_identifier), 
           newargs, self.config.get_code_object_identifier
           state,
           None,
           None,
           _function_setstate)
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417274250


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   i wasn't able to figure this out. any ideas?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377260012


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -78,6 +78,9 @@
 import warnings
 import weakref
 
+from apache_beam.internal.code_object_pickler import get_code_from_identifier

Review Comment:
   +1 to run the CP test, we can try that on your fork before merging. adding existing CloudPickle  test coverage can be a separate PR if we go that route. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377292871


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock

Review Comment:
   would it suffice to:
   
   ```
   def _custom_dynamic_function_reduce():
      ...
   
   if (<setting is enabled>):
      cloudpickle._dynamic_function_reduce = _custom_dynamic_function_reduce
   ```
   
   so that we:
   1) patch a single function
   2) don't pull a test dependency (unittest.mock)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3285685883

   Hi sorry I did not notice this. I will look at it today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377243995


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   > the first element that function_getnewargs returns is a string if the code object identifier was able to be created. in that case, we want to use make_function_from_identifier.
     
   Instead of modifying   `_function_getnewargs` could we instead have  the branching within `_dynamic_function_reduce`? It looks like `_function_getnewargs` is not used anywhere else. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377292871


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock

Review Comment:
   would it suffice to:
   
   ```
   def _patched_dynamic_function_reduce():
      ...
   
   if (<setting is enabled>):
      cloudpickle._dynamic_function_reduce = _patched_dynamic_function_reduce
   ```
   
   so that we:
   1) patch a single function
   2) don't pull a test dependency (unittest.mock)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2379922229


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock

Review Comment:
   Yes, I think we could forego the context manager and do the unconditional patching. I think that's sufficient for your usecase since for the lifetime of the process the setting is either enabled or disabled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2380091205


##########
sdks/python/apache_beam/internal/cloudpickle_pickler.py:
##########
@@ -29,16 +29,20 @@
 
 import base64
 import bz2
+import contextlib
 import io
 import logging
 import sys
 import threading
+from unittest import mock
 import zlib
 
 from apache_beam.internal.cloudpickle import cloudpickle
+from apache_beam.internal import code_object_pickler
 
 DEFAULT_CONFIG = cloudpickle.CloudPickleConfig(
-    skip_reset_dynamic_type_state=True)
+    skip_reset_dynamic_type_state=True,
+    enable_stable_function_identifiers=False)

Review Comment:
   please add description of the setting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416758809


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)

Review Comment:
   Can we replace this with pickler.loads(pickler.dumps(lambda x: x, enable_stable_code_identifier_pickling=True))?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3382379320

   @kristynsmith can you please check failing tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2433380786


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +302,48 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_stable_identifier_uses_current_code(self):
+    pickler.set_library('cloudpickle')
+
+    # Get original dynamic function
+    func_v1 = module_test.mutable_test_function()
+
+    pickled_stable = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=True)
+
+    pickled_frozen = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=False)
+
+    # Save original function for cleanup
+    original_function = module_test.mutable_test_function
+
+    try:
+      # Monkey patch: Replace the entire outer function with v2
+      code_v2 = """
+def mutable_test_function():

Review Comment:
   Replace with
   ```
         code_v2 = textwrap.dedent("""
         def mutable_test_function():
           def dynamic_function():
             return "version2"
   
         return dynamic_function
         """)
   ```
   
   Also `import textwrap` into the pickler_test.py file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3285055731

   Reminder, please take a look at this pr: @liferoad 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2380089150


##########
sdks/python/apache_beam/runners/pipeline_context.py:
##########
@@ -227,6 +227,7 @@ def __init__(
     self.iterable_state_write = iterable_state_write
     self._requirements = set(requirements)
     self.enable_best_effort_deterministic_pickling = False
+    self.enable_lambda_name_pickling = False

Review Comment:
   do we still need this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417046256


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -29,11 +29,27 @@
 from parameterized import param
 from parameterized import parameterized
 
+from apache_beam.internal import code_object_pickler
 from apache_beam.internal import module_test
 from apache_beam.internal import pickler
+from apache_beam.internal.cloudpickle import cloudpickle
+from apache_beam.internal.cloudpickle_pickler import dumps as cloudpickle_dumps
 from apache_beam.internal.pickler import dumps
 from apache_beam.internal.pickler import loads
 
+STABLE_CODE_IDENTIFIER_PICKLING_CONFIG = cloudpickle.CloudPickleConfig(

Review Comment:
   Remove this to line 51



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417047002


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickler.loads(pickler.dumps(lambda x: x, enable_stable_code_identifier_pickling=True))
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)
+
+  def test_disable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, False)

Review Comment:
   Replace with `pickled = pickler.loads(pickler.dumps(lambda x: x, enable_stable_code_identifier_pickling=False))`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2417048469


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   originally i had tried 
   ```
   def test_enable_lambda_name_pickling(self):
       pickler.set_library('cloudpickle')
       lambda_name = pickle(lambda x: x, enable_lambda_name=True)
       self.assertEqual(lambda_name, '__main__.PicklerTest.test_enable_lambda_name_pickling.__code__.co_consts[<lambda>]')
   ```
       
   and attempted to write a pickle() function that would return the pickled value but i wasn't able to get it to work. i could try this again though



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2430008904


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   The test is still failing. 
   
   ```
   # Get original dynamic function
   >     func_v1 = module_test.mutable_test_function()
   E     AttributeError: module 'apache_beam.internal.module_test' has no attribute 'mutable_test_function'
   ```
   
   Need to define mutable_test_function in https://github.com/apache/beam/blob/385271bea452dc3394dae3121035007167c6fb58/sdks/python/apache_beam/internal/module_test.py#L4
   
   ```
   def mutable_test_function():
       def dynamic_function():
           return "version1"
       return dynamic_function
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3422817022

   test results looks strange as if they ran on some other commit, let me close and reopenl.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed pull request #35904: Integrate lambda name pickling with Cloudpickle
URL: https://github.com/apache/beam/pull/35904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377243995


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   > the first element that function_getnewargs returns is a string if the code object identifier was able to be created. in that case, we want to use make_function_from_identifier.
    
   Instead of modifying   _function_getnewargs could we instead have the branching within _dynamic_function_reduce? It looks like _function_getnewargs is not used anywhere else.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2430008904


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   The test is still failing. 
   # Get original dynamic function
   >     func_v1 = module_test.mutable_test_function()
   E     AttributeError: module 'apache_beam.internal.module_test' has no attribute 'mutable_test_function'
   
   Need to define mutable_test_function in https://github.com/apache/beam/blob/385271bea452dc3394dae3121035007167c6fb58/sdks/python/apache_beam/internal/module_test.py#L4
   
   ```
   def mutable_test_function():
       def dynamic_function():
           return "version1"
       return dynamic_function
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2433380786


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +302,48 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_stable_identifier_uses_current_code(self):
+    pickler.set_library('cloudpickle')
+
+    # Get original dynamic function
+    func_v1 = module_test.mutable_test_function()
+
+    pickled_stable = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=True)
+
+    pickled_frozen = pickler.dumps(
+        func_v1, enable_stable_code_identifier_pickling=False)
+
+    # Save original function for cleanup
+    original_function = module_test.mutable_test_function
+
+    try:
+      # Monkey patch: Replace the entire outer function with v2
+      code_v2 = """
+def mutable_test_function():

Review Comment:
   Replace with
   ```
         code_v2 = textwrap.dedent("""
           def mutable_test_function():
             def dynamic_function():
               return "version2"
   
             return dynamic_function
         """)
   ```
   
   Also `import textwrap` into the pickler_test.py file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #35904:
URL: https://github.com/apache/beam/pull/35904#issuecomment-3427829649

   Thanks, @kristynsmith and @claudevdm .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #35904:
URL: https://github.com/apache/beam/pull/35904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2377243995


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   I am looking to see how to simplify your monkey patch, the below comment is concerning your original version, excluding your commits from the last iteration: (https://github.com/apache/beam/compare/master...tvalentyn:subset1)
   
   > the first element that function_getnewargs returns is a string if the code object identifier was able to be created. in that case, we want to use make_function_from_identifier.
     
   Instead of modifying   `_function_getnewargs` could we instead have  the branching within `_dynamic_function_reduce`? It looks like `_function_getnewargs` is not used anywhere else. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2379924304


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   i think this should simply things quite a bit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416765628


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   What is this test trying to verify? Is it that pickling/unpickling works?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2416930104


##########
sdks/python/apache_beam/internal/pickler_test.py:
##########
@@ -302,6 +318,18 @@ def test_disable_best_effort_determinism(self):
         dumps(set1, enable_best_effort_determinism=False),
         dumps(set2, enable_best_effort_determinism=False))
 
+  def test_enable_stable_code_identifier_pickling(self):
+    pickler.set_library('cloudpickle')
+    pickled = pickle_depickle(lambda x: x, True)
+    pickled_type = type(pickled)
+    self.assertIsInstance(pickled, pickled_type)

Review Comment:
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Integrate lambda name pickling with Cloudpickle [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35904:
URL: https://github.com/apache/beam/pull/35904#discussion_r2350255226


##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1326,7 +1343,12 @@ def dump(self, obj):
         raise
 
   def __init__(
-      self, file, protocol=None, buffer_callback=None, config=DEFAULT_CONFIG):
+      self,
+      file,
+      protocol=None,
+      buffer_callback=None,
+      config=DEFAULT_CONFIG,
+      enable_lambda_name=False):

Review Comment:
   1. If we must pass configs to cloudpickle,  we should adjust settings via the `config` param instead of adding new params here.
   
   2. We should try to think how this change could potentially be ported into cloudpickle. Ideally, we wouldn't have the imports like
    
   ``` 
   from apache_beam.internal.code_object_pickler import get_code_from_identifier
   from apache_beam.internal.code_object_pickler import get_code_object_identifier
   ```
   
   possible approaches:
   
   1) We monkey-patch `cloudpickle._make_function` from `apache_beam/internal/cloudpickle_pickler.py`, when a certain setting is set.
   
   2) We pass callables implementing some functionality in the config, instead of monkey-patching, for example:
   `config.custom_make_function=apache_beam/internal/cloudpickle_pickler.make_function`
   
   3) We move necessary functionality into cloudpickle.py, and then change behaviors based on config.enable_some_cloudpickle_setting
   
    If the customized pickling functionality could be useful for other cloudpickle users, then i'd opt for 3. If other cloudpickle user might have a different implementation of their own for make_function, then 2. 1 is a quick-and-dirty option we could choose to avoid overthinking for now.



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1336,6 +1358,7 @@ def __init__(
     self.globals_ref = {}
     self.proto = int(protocol)
     self.config = config
+    self.enable_lambda_name = enable_lambda_name

Review Comment:
   can we think of a better name for this config that would make it easier to communicate to an unfamiliar reader what pickling behavior is being customized? Do you have any other naming suggestions? How would you describe this setting in a docstring?
   



##########
sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:
##########
@@ -1266,7 +1274,11 @@ def _dynamic_function_reduce(self, func):
     """Reduce a function that is not pickleable via attribute lookup."""
     newargs = self._function_getnewargs(func)
     state = _function_getstate(func)
-    return (_make_function, newargs, state, None, None, _function_setstate)
+    if type(newargs[0]) == str:

Review Comment:
   1. for my education what's the nature of this condition (why this particular condition?) `    if type(newargs[0]) == str:`
   2. I think this change should be configurable (when certain setting is enabled behavior is changed, otherwise keep the original behavior).
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org