You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/30 04:09:43 UTC
[GitHub] [beam] cozos commented on issue #21598: Beam worker closing gRPC connection with many workers and large shuffle sizes
cozos commented on issue #21598:
URL: https://github.com/apache/beam/issues/21598#issuecomment-1331619714
Also running into this with the Spark RDD Runner on the Python SDK Harness:
```
22/11/30 03:35:04 INFO Executor: Finished task 505.0 in stage 8.0 (TID 1145). 18572 bytes result sent to driver
22/11/30 03:35:55 INFO Executor: Finished task 434.0 in stage 8.0 (TID 1074). 18572 bytes result sent to driver
E1130 03:36:33.423173032 2378 chttp2_transport.cc:1016] ipv4:127.0.0.1:38291: Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings". Current keepalive time (before throttling): 20000ms
22/11/30 03:36:33 ERROR py:641: Failed to read inputs in the data plane.
Traceback (most recent call last):
File "/databricks/python3/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py", line 634, in _read_inputs
for elements in elements_iterator:
File "/databricks/python3/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
return self._next()
File "/databricks/python3/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:38291 {grpc_message:"Socket closed", grpc_status:14, created_time:"2022-11-30T03:36:33.423534531+00:00"}"
> Traceback (most recent call last):
File "/databricks/python3/lib/python3.7/site-packages/apache_beam/runners/worker/data_plane.py", line 634, in _read_inputs
for elements in elements_iterator:
File "/databricks/python3/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
return self._next()
File "/databricks/python3/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:38291 {grpc_message:"Socket closed", grpc_status:14, created_time:"2022-11-30T03:36:33.423534531+00:00"}"
```
I do have a shuffle/GroupBy, but I don't understand why that would cause this. As I understand it the SDK Harness is only used to execute DoFns, and the GroupBy/shuffle is done on the Spark side, which shouldn't affect the SDK Harness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org