You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "parveensania (via GitHub)" <gi...@apache.org> on 2026/08/05 21:35:11 UTC
[PR] Make primary channel failover timeout configurable [beam]
parveensania opened a new pull request, #39646:
URL: https://github.com/apache/beam/pull/39646
This PR makes the timeout threshold before the `FailoverChannel` switches to a secondary (fallback) channel configurable.
Previously, this wait time was hardcoded to 10 seconds. Making it configurable allows for easier tweaking and better control over failover behavior for directpath enabled jobs. The PR also increases the default from 10s to 15s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on PR #39646:
URL: https://github.com/apache/beam/pull/39646#issuecomment-5197684456
@arunpandianp could you review this? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3724610845
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannel.java:
##########
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+import java.net.http.WebSocket.Listener;
Review Comment:
Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3807869864
##########
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannelTest.java:
##########
@@ -283,11 +284,110 @@ public void testStateFallbackAfterPrimaryNotReady() {
// Within 10 seconds: grace period not elapsed, routes to primary.
failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
verify(mockChannel).newCall(any(), any());
+ // Fallback must not have been used yet.
+ verify(mockFallbackChannel, never()).newCall(any(), any());
// After 10 seconds: routes to fallback.
time.addAndGet(TimeUnit.SECONDS.toNanos(11));
failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
verify(mockFallbackChannel).newCall(any(), any());
+ // Primary must not have been used for this call (still only 1 invocation).
+ verify(mockChannel, org.mockito.Mockito.times(1)).newCall(any(), any());
Review Comment:
Done
##########
runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto:
##########
@@ -1018,6 +1018,10 @@ message UserWorkerRunnerV1Settings {
optional int64 max_cached_entry_bytes = 5 [default = -1];
+ // Time to wait before switching to fallback connectivity if primary is not ready.
+ // Only used if direcpath is enabled for the job. Default is 15 seconds.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on PR #39646:
URL: https://github.com/apache/beam/pull/39646#issuecomment-5198640024
Run RAT PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle merged PR #39646:
URL: https://github.com/apache/beam/pull/39646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3802042254
##########
runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto:
##########
@@ -1018,6 +1018,10 @@ message UserWorkerRunnerV1Settings {
optional int64 max_cached_entry_bytes = 5 [default = -1];
+ // Time to wait before switching to fallback connectivity if primary is not ready.
+ // Only used if direcpath is enabled for the job. Default is 15 seconds.
+ optional int64 directpath_primary_not_ready_wait_nanos = 6 [default = 15000000000];
Review Comment:
let's use millis instead of nanos for proto, it's overly specific and too easy to get wrong I think.
##########
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannelTest.java:
##########
@@ -283,11 +284,110 @@ public void testStateFallbackAfterPrimaryNotReady() {
// Within 10 seconds: grace period not elapsed, routes to primary.
failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
verify(mockChannel).newCall(any(), any());
+ // Fallback must not have been used yet.
+ verify(mockFallbackChannel, never()).newCall(any(), any());
// After 10 seconds: routes to fallback.
time.addAndGet(TimeUnit.SECONDS.toNanos(11));
failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
verify(mockFallbackChannel).newCall(any(), any());
+ // Primary must not have been used for this call (still only 1 invocation).
+ verify(mockChannel, org.mockito.Mockito.times(1)).newCall(any(), any());
Review Comment:
nit: static import of times to cleanup these call sites
##########
runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto:
##########
@@ -1018,6 +1018,10 @@ message UserWorkerRunnerV1Settings {
optional int64 max_cached_entry_bytes = 5 [default = -1];
+ // Time to wait before switching to fallback connectivity if primary is not ready.
+ // Only used if direcpath is enabled for the job. Default is 15 seconds.
Review Comment:
spelling: direcpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on PR #39646:
URL: https://github.com/apache/beam/pull/39646#issuecomment-5198577368
Run Java Dataflow Non-portable Worker PreCommit
Run RAT PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3797664473
##########
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannelTest.java:
##########
@@ -290,6 +291,91 @@ public void testStateFallbackAfterPrimaryNotReady() {
verify(mockFallbackChannel).newCall(any(), any());
}
+ @Test
+ public void testTimeoutThresholdDecreaseTriggersFallbackEarlier() {
+ ManagedChannel mockChannel = mock(ManagedChannel.class);
+ ManagedChannel mockFallbackChannel = mock(ManagedChannel.class);
+ when(mockChannel.newCall(any(), any())).thenReturn(mock(ClientCall.class));
+ when(mockFallbackChannel.newCall(any(), any())).thenReturn(mock(ClientCall.class));
+ // Simulate primary being TRANSIENT_FAILURE from the start.
+ when(mockChannel.getState(false)).thenReturn(ConnectivityState.TRANSIENT_FAILURE);
+
+ AtomicLong time = new AtomicLong(0);
+ // Start with 10s timeout
+ AtomicLong timeoutThreshold = new AtomicLong(TimeUnit.SECONDS.toNanos(10));
+
+ // Constructor seeds timer at time=0.
+ FailoverChannel failoverChannel =
+ FailoverChannel.forTest(
+ mockChannel, mockFallbackChannel, null, time::get, 0L, timeoutThreshold::get);
+
+ // Call at time=0. elapsed 0 <= 10s -> false.
+ failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
+ verify(mockChannel).newCall(any(), any());
Review Comment:
Done. Updated the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3724612694
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java:
##########
@@ -828,6 +829,7 @@ private static ChannelCache createChannelCache(
DataflowWorkerHarnessOptions workerOptions,
ComputationConfig.Fetcher configFetcher,
GrpcDispatcherClient dispatcherClient) {
+ AtomicLong directpathPrimaryNotReadyWaitNanos = new AtomicLong(TimeUnit.SECONDS.toNanos(15));
Review Comment:
That's a good point. Made the suggested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3807872622
##########
runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto:
##########
@@ -1018,6 +1018,10 @@ message UserWorkerRunnerV1Settings {
optional int64 max_cached_entry_bytes = 5 [default = -1];
+ // Time to wait before switching to fallback connectivity if primary is not ready.
+ // Only used if direcpath is enabled for the job. Default is 15 seconds.
+ optional int64 directpath_primary_not_ready_wait_nanos = 6 [default = 15000000000];
Review Comment:
Changed the unit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39646:
URL: https://github.com/apache/beam/pull/39646#issuecomment-5280375901
Reminder, please take a look at this pr: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "parveensania (via GitHub)" <gi...@apache.org>.
parveensania commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3797662705
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannel.java:
##########
@@ -426,11 +445,12 @@ private void onPrimaryStateChanged() {
if (newState == ConnectivityState.READY || newState == ConnectivityState.IDLE) {
if (state.markPrimaryReady()) {
LOG.info(
- "[channel-{}] Primary channel recovered; switching back from fallback.", channelId);
+ "[channel-{}] Primary channel observed healthy during state change registration; switching back from fallback.",
Review Comment:
This was supposed to be "Primary channel observed healthy during state change notification". Had made a typo. Thanks for catching it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3724565001
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java:
##########
@@ -828,6 +829,7 @@ private static ChannelCache createChannelCache(
DataflowWorkerHarnessOptions workerOptions,
ComputationConfig.Fetcher configFetcher,
GrpcDispatcherClient dispatcherClient) {
+ AtomicLong directpathPrimaryNotReadyWaitNanos = new AtomicLong(TimeUnit.SECONDS.toNanos(15));
Review Comment:
make it a static member of StreamingDataflowWorker? So we can get a hold on this if needed (eg: to print in status pages).
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannel.java:
##########
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+import java.net.http.WebSocket.Listener;
Review Comment:
looks unused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39646:
URL: https://github.com/apache/beam/pull/39646#issuecomment-5199410689
Assigning reviewers:
R: @Abacn added as fallback since no labels match configuration
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make primary channel failover timeout configurable [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #39646:
URL: https://github.com/apache/beam/pull/39646#discussion_r3728059323
##########
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannelTest.java:
##########
@@ -290,6 +291,91 @@ public void testStateFallbackAfterPrimaryNotReady() {
verify(mockFallbackChannel).newCall(any(), any());
}
+ @Test
+ public void testTimeoutThresholdDecreaseTriggersFallbackEarlier() {
+ ManagedChannel mockChannel = mock(ManagedChannel.class);
+ ManagedChannel mockFallbackChannel = mock(ManagedChannel.class);
+ when(mockChannel.newCall(any(), any())).thenReturn(mock(ClientCall.class));
+ when(mockFallbackChannel.newCall(any(), any())).thenReturn(mock(ClientCall.class));
+ // Simulate primary being TRANSIENT_FAILURE from the start.
+ when(mockChannel.getState(false)).thenReturn(ConnectivityState.TRANSIENT_FAILURE);
+
+ AtomicLong time = new AtomicLong(0);
+ // Start with 10s timeout
+ AtomicLong timeoutThreshold = new AtomicLong(TimeUnit.SECONDS.toNanos(10));
+
+ // Constructor seeds timer at time=0.
+ FailoverChannel failoverChannel =
+ FailoverChannel.forTest(
+ mockChannel, mockFallbackChannel, null, time::get, 0L, timeoutThreshold::get);
+
+ // Call at time=0. elapsed 0 <= 10s -> false.
+ failoverChannel.newCall(methodDescriptor, CallOptions.DEFAULT);
+ verify(mockChannel).newCall(any(), any());
Review Comment:
can you verify no new interactions on the channel not used for all of these?
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannel.java:
##########
@@ -426,11 +445,12 @@ private void onPrimaryStateChanged() {
if (newState == ConnectivityState.READY || newState == ConnectivityState.IDLE) {
if (state.markPrimaryReady()) {
LOG.info(
- "[channel-{}] Primary channel recovered; switching back from fallback.", channelId);
+ "[channel-{}] Primary channel observed healthy during state change registration; switching back from fallback.",
Review Comment:
this isn't during registration but after notification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org