You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2024/08/02 15:28:07 UTC
(beam) branch users/damccorm/pickling-across-processes created (now 3ca14d7c7e9)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a change to branch users/damccorm/pickling-across-processes
in repository https://gitbox.apache.org/repos/asf/beam.git
at 3ca14d7c7e9 Add warning + doc callout when encountering ri pickling errors
This branch includes the following new commits:
new 3ca14d7c7e9 Add warning + doc callout when encountering ri pickling errors
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(beam) 01/01: Add warning + doc callout when encountering ri pickling errors
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/pickling-across-processes
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 3ca14d7c7e90028461ea47f6f6e8520e78aeae45
Author: Danny McCormick <da...@google.com>
AuthorDate: Fri Aug 2 17:27:45 2024 +0200
Add warning + doc callout when encountering ri pickling errors
---
sdks/python/apache_beam/ml/inference/base.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/sdks/python/apache_beam/ml/inference/base.py b/sdks/python/apache_beam/ml/inference/base.py
index 401b57fdb80..29a568def07 100644
--- a/sdks/python/apache_beam/ml/inference/base.py
+++ b/sdks/python/apache_beam/ml/inference/base.py
@@ -1586,6 +1586,15 @@ class _RunInferenceDoFn(beam.DoFn, Generic[ExampleT, PredictionT]):
except BaseException as e:
if self._metrics_collector:
self._metrics_collector.failed_batches_counter.inc()
+ if (e is pickle.PickleError and
+ self._model_handler.share_model_across_processes()):
+ raise TypeError(
+ 'Pickling error encountered while running inference. '
+ 'This may be caused by trying to send unpickleable '
+ 'data to a model which is shared across processes. '
+ 'For more information, see '
+ 'https://beam.apache.org/documentation/ml/large-language-modeling/#pickling-errors' # pylint: disable=line-too-long
+ ) from e
raise e
predictions = list(result_generator)