You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Juan Romero <js...@gmail.com> on 2024/02/24 02:14:04 UTC
Problem with pikcler
Hi guys.
Currently i have the class SaptmCustomerUpsertPipeline that inherited from
the GenericPipeline class. Locally the pipeline works fine, but when i try
to run it in dataflow i got this error:
severity: "ERROR"
textPayload: "Error message from worker: generic::unknown: Traceback (most
recent call last): File
"/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
line 285, in loads return dill.loads(s) File
"/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
return load(file, ignore, **kwds) File
"/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
return Unpickler(file, ignore=ignore, **kwds).load() File
"/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
obj = StockUnpickler.load(self) File
"/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in
find_class return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on
<module 'apache_beam.runners.worker.sdk_worker_main' from
'/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
Seems that there is an error in the serialization of the child. class.
Someone has faced with this problem? . The pipeline works fine with the
direct runner.
Re: Problem with pikcler
Posted by Juan Romero <js...@gmail.com>.
Thanks!!’
On Sat, 24 Feb 2024 at 12:23 PM Valentyn Tymofieiev via user <
user@beam.apache.org> wrote:
> You might need to pass the --save_main_session pipeline option. See:
> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
>
> Alternatively, structure your pipeline as a package:
> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython
>
>
> On Fri, Feb 23, 2024 at 6:14 PM Juan Romero <js...@gmail.com> wrote:
>
>> Hi guys.
>>
>> Currently i have the class SaptmCustomerUpsertPipeline that inherited
>> from the GenericPipeline class. Locally the pipeline works fine, but when i
>> try to run it in dataflow i got this error:
>>
>> severity: "ERROR"
>> textPayload: "Error message from worker: generic::unknown: Traceback
>> (most recent call last): File
>> "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
>> line 285, in loads return dill.loads(s) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
>> return load(file, ignore, **kwds) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
>> return Unpickler(file, ignore=ignore, **kwds).load() File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
>> obj = StockUnpickler.load(self) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in
>> find_class return StockUnpickler.find_class(self, module, name)
>> AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on
>> <module 'apache_beam.runners.worker.sdk_worker_main' from
>> '/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
>>
>> Seems that there is an error in the serialization of the child. class.
>> Someone has faced with this problem? . The pipeline works fine with the
>> direct runner.
>>
>>
Re: Problem with pikcler
Posted by Valentyn Tymofieiev via user <us...@beam.apache.org>.
You might need to pass the --save_main_session pipeline option. See:
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
Alternatively, structure your pipeline as a package:
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython
On Fri, Feb 23, 2024 at 6:14 PM Juan Romero <js...@gmail.com> wrote:
> Hi guys.
>
> Currently i have the class SaptmCustomerUpsertPipeline that inherited from
> the GenericPipeline class. Locally the pipeline works fine, but when i try
> to run it in dataflow i got this error:
>
> severity: "ERROR"
> textPayload: "Error message from worker: generic::unknown: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
> line 285, in loads return dill.loads(s) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
> return load(file, ignore, **kwds) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
> return Unpickler(file, ignore=ignore, **kwds).load() File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
> obj = StockUnpickler.load(self) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in
> find_class return StockUnpickler.find_class(self, module, name)
> AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on
> <module 'apache_beam.runners.worker.sdk_worker_main' from
> '/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
>
> Seems that there is an error in the serialization of the child. class.
> Someone has faced with this problem? . The pipeline works fine with the
> direct runner.
>
>