You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2023/07/25 14:12:01 UTC

[GitHub] [beam] riteshghorse opened a new issue, #27653: [Bug]: Pickling error when running pipelines with TFModelHandlerTensor on Python 3.9.2

riteshghorse opened a new issue, #27653:
URL: https://github.com/apache/beam/issues/27653

   ### What happened?
   
   Running python pipelines with `TFModelHandlerTensor` when used with `load_model_args` got some pickling issues on Direct Runner. Haven't verified on other runners
   ```
     File "/home/student_00_40bc0e514d9c/devrel-demos/data-analytics/beam_ml_toxicity_in_gaming/part1.py", line 141, in <module>
       run(known_args.project_id, known_args.gaming_loc, known_args.movie_loc, pipeline_args)
     File "/home/student_00_40bc0e514d9c/devrel-demos/data-analytics/beam_ml_toxicity_in_gaming/part1.py", line 117, in run
       _ = (not_filter
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/pipeline.py", line 600, in __exit__
       self.result = self.run()
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/pipeline.py", line 574, in run
       pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/pickler.py", line 61, in dump_session
       return desired_pickle_lib.dump_session(file_path)
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", line 439, in dump_session
       dill.dump_session(file_path)
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py", line 351, in dump_session
       pickler.dump(main)
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py", line 445, in dump
       StockPickler.dump(self, obj)
     File "/usr/lib/python3.9/pickle.py", line 487, in dump
       self.save(obj)
     File "/usr/lib/python3.9/pickle.py", line 560, in save
       f(self, obj)  # Call unbound method with explicit self
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", line 301, in save_module
       return old_save_module(pickler, obj)
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py", line 1294, in save_module
       pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
     File "/usr/lib/python3.9/pickle.py", line 717, in save_reduce
       save(state)
     File "/usr/lib/python3.9/pickle.py", line 560, in save
       f(self, obj)  # Call unbound method with explicit self
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", line 349, in new_save_module_dict
       return old_save_module_dict(pickler, obj)
     File "/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py", line 912, in save_module_dict
       StockPickler.save_dict(pickler, obj)
     File "/usr/lib/python3.9/pickle.py", line 971, in save_dict
       self._batch_setitems(obj.items())
     File "/usr/lib/python3.9/pickle.py", line 997, in _batch_setitems
       save(v)
     File "/usr/lib/python3.9/pickle.py", line 578, in save
       rv = reduce(self.proto)
   TypeError: cannot pickle 'LazyLoader' object
   ``` 
   
   ### Issue Priority
   
   Priority: 3 (minor)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] riteshghorse commented on issue #27653: [Bug]: Pickling error when running pipelines with TFModelHandlerTensor on Python 3.9.2

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #27653:
URL: https://github.com/apache/beam/issues/27653#issuecomment-1649926057

   Upgrading the python's version to 3.9.13 solved it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] riteshghorse closed issue #27653: [Bug]: Pickling error when running pipelines with TFModelHandlerTensor on Python 3.9.2

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse closed issue #27653: [Bug]: Pickling error when running pipelines with TFModelHandlerTensor on Python 3.9.2
URL: https://github.com/apache/beam/issues/27653


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org