You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "kristynsmith (via GitHub)" <gi...@apache.org> on 2025/06/03 17:51:40 UTC

[PR] Move non-dill specific code out of dill_pickler.py [beam]

kristynsmith opened a new pull request, #35139:
URL: https://github.com/apache/beam/pull/35139

   Move non-dill specific code from dill_pickler.py into new file consistent_pickle.py
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2949889922

   LGTM. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2949652537

   Run Puthon_Dataframes PreCommit 3.11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "AdrS (via GitHub)" <gi...@apache.org>.
AdrS commented on code in PR #35139:
URL: https://github.com/apache/beam/pull/35139#discussion_r2127186816


##########
sdks/python/apache_beam/internal/code_object_pickler.py:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Customizations to how Python code objects are pickled.
+
+This module provides functions to pickle code objects, especially lambdas, in a
+consistent way. It addresses issues with non-deterministic pickling by creating
+stable references to code objects using their qualified names and bytecode

Review Comment:
   This comment is only relevant to the internal google3 version of the code. Please remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2949418175

   Assigning reviewers:
   
   R: @shunping for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2941319082

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2949413378

   assign set of reviewers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #35139:
URL: https://github.com/apache/beam/pull/35139#discussion_r2132317529


##########
sdks/python/apache_beam/internal/code_object_pickler.py:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Customizations to how Python code objects are pickled.
+
+This module provides functions to pickle code objects, especially lambdas, in a
+consistent way. It addresses issues with non-deterministic pickling by creating
+stable references to code objects using their qualified names and bytecode

Review Comment:
   I think @AdrS meant to remove the selected lines not the apache license at the beginning... 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2946090901

   Run Python_Coverage PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2949763122

   Run Python_Coverage PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2945978984

   Run Python_Coverage PreCommit
   Run preCommitPyCoverage
   Run preCommitPyCoverage PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2945770882

   Run Python_Coverage PreCommit
   Run Python_Runners PreCommit
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #35139:
URL: https://github.com/apache/beam/pull/35139#discussion_r2132317529


##########
sdks/python/apache_beam/internal/code_object_pickler.py:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Customizations to how Python code objects are pickled.
+
+This module provides functions to pickle code objects, especially lambdas, in a
+consistent way. It addresses issues with non-deterministic pickling by creating
+stable references to code objects using their qualified names and bytecode

Review Comment:
   I think @AdrS meant to remove the selected lines not the apache license at the beginning... 
   
   Without apache license at the beginning, PreCommit RAT will fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad merged PR #35139:
URL: https://github.com/apache/beam/pull/35139


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Move non-dill specific code out of dill_pickler.py [beam]

Posted by "kristynsmith (via GitHub)" <gi...@apache.org>.
kristynsmith commented on PR #35139:
URL: https://github.com/apache/beam/pull/35139#issuecomment-2945795261

   Run Python_Runners PreCommit 3.9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org