Beam on Flink: GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"

[Janek Bevendorff <ja...@uni-weimar.de>]

Hi,

There are multiple issue reports about this or similar issues on 
GitHub/Jira but all of them without any proper solution, so maybe you 
can help me.

I am running Beam on Flink (using the Portable runner via Beam's Flink 
job server) and when something takes a bit longer than expected or the 
shuffle size gets a bit larger, my workers keep failing randomly with 
the following error:

E0922 08:50:52.814447061     222 chttp2_transport.cc:1167] Received a 
GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to 
"too_many_pings"

I have already tried adding

("grpc.http2.max_pings_without_data", 0),
("grpc.http2.max_ping_strikes", 0)

to DEFAULT_OPTIONS in 
sdks/python/apache_beam/runners/worker/channel_factory.py, but without 
success. Are there any other places where gRPC connections are 
established that need these extra options? Are there any other options 
that I overlooked?

The most relevant (unsolved) issue report is probably this one here: 
https://github.com/apache/beam/issues/21598

This issue is pretty serious, since it pretty much prevents me from 
running jobs with more than a handful of workers or large data.

Many thanks
Janek