You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Jeff Webb (Jira)" <ji...@apache.org> on 2021/09/14 22:48:00 UTC
[jira] [Updated] (BEAM-9249) Python SDK DirectRunner with
multiprocessing fails with grpc.RPCError: 'Received message larger than
max'
[ https://issues.apache.org/jira/browse/BEAM-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Webb updated BEAM-9249:
----------------------------
Resolution: Won't Fix
Status: Resolved (was: Triage Needed)
old issue - resolving
> Python SDK DirectRunner with multiprocessing fails with grpc.RPCError: 'Received message larger than max'
> ---------------------------------------------------------------------------------------------------------
>
> Key: BEAM-9249
> URL: https://issues.apache.org/jira/browse/BEAM-9249
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.19.0
> Reporter: Enis Nazif
> Priority: P3
>
> When running a pipeline using the Python SDK, DirectRunner, and a direct_running_mode of 'multi_processing' or 'multi_threading', the job fails with
> Reproducible example:
> {code:java}
> import apache_beam as beam
> p = beam.Pipeline(options=PipelineOptions(direct_running_mode='multi_processing', num_direct_runners=4)) | beam.Create([i for i in range(1000000)])
> p.pipeline.run(){code}
>
> Output:
> {code:java}
> Traceback (most recent call last):Traceback (most recent call last): File "/opt/man/releases/python-medusa/36-1/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/opt/man/releases/python-medusa/36-1/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/users/is/enazif/beam/sdks/python/apache_beam/runners/worker/sdk_worker_main.py", line 249, in <module> main(sys.argv) File "/users/is/enazif/beam/sdks/python/apache_beam/runners/worker/sdk_worker_main.py", line 160, in main sdk_pipeline_options.view_as(ProfilingOptions)) File "/users/is/enazif/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 137, in run for work_request in control_stub.Control(get_responses()): File "/users/is/enazif/pyenvs/04-02-2020-1/lib/python3.6/site-packages/grpcio-1.18.0+ahl1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 364, in __next__ return self._next() File "/users/is/enazif/pyenvs/04-02-2020-1/lib/python3.6/site-packages/grpcio-1.18.0+ahl1-py3.6-linux-x86_64.egg/grpc/_channel.py", line 358, in _next raise selfgrpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (4806980 vs. 4194304)" debug_error_string = "{"created":"@1580828874.271749579","description":"Received message larger than max (4806980 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}">Exception in thread run_worker:Traceback (most recent call last): File "/opt/man/releases/python-medusa/36-1/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/man/releases/python-medusa/36-1/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/users/is/enazif/beam/sdks/python/apache_beam/runners/portability/local_job_service.py", line 218, in run 'Worker subprocess exited with return code %s' % p.returncode)RuntimeError: Worker subprocess exited with return code 1
> {code}
>
> This seems to be due to the fact that we're setting grpc.max_receive_message_length and grpc.max_send_message_length on the GRPC server side, but not on the channel side ([https://github.com/apache/beam/blob/7547ac6b273e6e2ffe7d69775606e14c0fd455b2/sdks/python/apache_beam/runners/worker/channel_factory.py#L28)|https://github.com/apache/beam/blob/7547ac6b273e6e2ffe7d69775606e14c0fd455b2/sdks/python/apache_beam/runners/worker/channel_factory.py#L28]
> Adding grpc.max_receive_message_length and grpc.max_send_message_length to the default options here fixes the issue.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)