You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "m-trieu (via GitHub)" <gi...@apache.org> on 2025/03/20 21:24:42 UTC
[PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
m-trieu opened a new pull request, #34367:
URL: https://github.com/apache/beam/pull/34367
GetWorkStream was previously terminating due to `DEADLINE_EXCEEDED` status leading to stream breaks and unnecessary retry spikes in the streaming backend.
- Add GetWorkStream management loop that gracefully terminates and restarts the stream
- Extend deadlines of direct streams to 1hr for less churn in restarting the streams internally
- Add worker token to stream manager thread name
R: @scwhittle @acrites
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2027947162
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
sgtm will draft this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025700039
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
we can expose a boolean to the constructor that will do this internally or allow the caller to manage the restarts (like in the case of `StreamPool`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073841688
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -120,21 +147,23 @@ protected AbstractWindmillStream(
this.isHealthCheckScheduled = new AtomicBoolean(false);
this.finishLatch = new CountDownLatch(1);
this.logger = logger;
+ this.sleeper = Sleeper.DEFAULT;
+ this.debugMetrics = StreamDebugMetrics.create();
+ this.streamTTL = streamTTL;
this.requestObserver =
new ResettableThrowingStreamObserver<>(
() ->
streamObserverFactory.from(
clientFactory,
new AbstractWindmillStream<RequestT, ResponseT>.ResponseObserver()),
logger);
- this.sleeper = Sleeper.DEFAULT;
- this.debugMetrics = StreamDebugMetrics.create();
}
- private static String createThreadName(String streamType, String backendWorkerToken) {
+ private static String createThreadName(
+ String threadType, String streamType, String backendWorkerToken) {
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023703078
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
+ }
+
+ // If graceful termination is unsuccessful, forcefully shutdown.
+ if (!newStream.awaitTermination(30, TimeUnit.SECONDS)) {
+ newStream.shutdown();
+ }
+
+ } catch (InterruptedException e) {
+ // continue until !isRunning.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025711489
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
`halfClose()` does share synchronization w/ `trySend()/send()` which can block indefinitely if its waiting for the underlying `StreamObserver.isReady`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-3124389103
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2045684742
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -195,6 +228,12 @@ private void startStream() {
onNewStream();
if (clientClosed) {
halfClose();
+ } else {
+ // If the stream is alive, schedule a future restart. We also attempt to cancel any
+ // pending restarts that have not run since they are used for preventing
+ // DEADLINE_EXCEEDED. Everytime the stream restarts, this deadline restarts so we want
+ // the stream restart to run whenever we restart the stream + streamTTL.
+ scheduleRestart();
Review Comment:
Would this be an appropriate place to call `requestObserver.startAsyncStreamCloser()`? It feels like this is the point where we know the stream got created so we want to close it eventually (as opposed to other code paths here in `startStream` where we might give up before actually creating the stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2891019799
Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025711489
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
`halfClose()` does share synchronization w/ `trySend()/send()` which can block indefinitely if its waiting for the underlying `StreamObserver.isReady`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023653864
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -88,19 +105,24 @@ private WindmillStreamSender(
streamingEngineStreamFactory.createDirectCommitWorkStream(
connection, streamingEngineThrottleTimers.commitWorkThrottleTimer());
this.workCommitter = workCommitterFactory.apply(commitWorkStream);
- this.getWorkStream =
- streamingEngineStreamFactory.createDirectGetWorkStream(
- connection,
- withRequestBudget(getWorkRequest, getWorkBudget.get()),
- streamingEngineThrottleTimers.getWorkThrottleTimer(),
- FixedStreamHeartbeatSender.create(getDataStream),
- getDataClientFactory.apply(getDataStream),
- workCommitter,
- workItemScheduler);
+ this.activeGetWorkStream = new AtomicReference<>();
+ this.getWorkStreamFactory =
+ () ->
+ streamingEngineStreamFactory.createDirectGetWorkStream(
+ connection,
+ withRequestBudget(getWorkRequest, getWorkBudget.get()),
+ streamingEngineThrottleTimers.getWorkThrottleTimer(),
+ FixedStreamHeartbeatSender.create(getDataStream),
+ getDataClientFactory.apply(getDataStream),
+ workCommitter,
+ workItemScheduler);
// 3 threads, 1 for each stream type (GetWork, GetData, CommitWork).
this.streamStarter =
Executors.newFixedThreadPool(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073648381
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -161,14 +190,15 @@ protected final synchronized boolean trySend(RequestT request)
try {
requestObserver.onNext(request);
return true;
- } catch (ResettableThrowingStreamObserver.StreamClosedException e) {
+ } catch (StreamClosedException e) {
// Stream was broken, requests may be retried when stream is reopened.
}
return false;
}
@Override
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023655268
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -126,35 +148,49 @@ private static GetWorkRequest withRequestBudget(GetWorkRequest request, GetWorkB
}
synchronized void start() {
- if (!started.get()) {
+ if (isRunning.compareAndSet(false, true)) {
checkState(!streamStarter.isShutdown(), "WindmillStreamSender has already been shutdown.");
-
// Start these 3 streams in parallel since they each may perform blocking IO.
+ CountDownLatch waitForInitialStream = new CountDownLatch(1);
+ streamStarter.execute(() -> getWorkStreamLoop(waitForInitialStream));
CompletableFuture.allOf(
- CompletableFuture.runAsync(getWorkStream::start, streamStarter),
CompletableFuture.runAsync(getDataStream::start, streamStarter),
CompletableFuture.runAsync(commitWorkStream::start, streamStarter))
.join();
+ try {
+ waitForInitialStream.await();
+ } catch (InterruptedException e) {
+ close();
+ LOG.error("GetWorkStream to {} was never able to start.", backendWorkerToken);
+ throw new IllegalStateException("GetWorkStream unable to start aborting.", e);
+ }
workCommitter.start();
- started.set(true);
}
}
@Override
public synchronized void close() {
+ isRunning.set(false);
streamStarter.shutdownNow();
- getWorkStream.shutdown();
getDataStream.shutdown();
workCommitter.stop();
commitWorkStream.shutdown();
}
@Override
public void setBudget(long items, long bytes) {
- GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
- getWorkBudget.set(budget);
- if (started.get()) {
- getWorkStream.setBudget(budget);
+ synchronized (activeGetWorkStream) {
+ GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
Review Comment:
moved the build out, i think setting it should still hold the lock since (from other comment) -> setting the budget on a stream can affect how many items/bytes we fetch in the header (and subsequent extension calls).
wanted to simplify the stream mechanics by preventing setBudget() from being called mid stream creation. That way either we start the stream first, and then call set budget which will update the internal stream budget and send an extension (possibly) OR we set the budget first, and start the stream w/ the new budget.
wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023745273
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/observers/DirectStreamObserver.java:
##########
@@ -217,11 +217,11 @@ public void terminate(Throwable terminationException) {
}
private String constructStreamCancelledErrorMessage(long totalSecondsWaited) {
- return deadlineSeconds > 0
+ return inactivityTimeout > 0
? "Waited "
+ totalSecondsWaited
+ "s which exceeds given deadline of "
- + deadlineSeconds
+ + inactivityTimeout
+ "s for the outboundObserver to become ready meaning "
+ "that the stream deadline was not respected."
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025699431
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
how about adding an internal mechanism (maybe via another executor to make it async if we dont want the old stream closing to block the new stream creation/stream restart).
Where we add this workflow:
- create stream
- start stream
- wait for termination based on a TTL/timeout
- loop while(stream.isRunning).
we can add this in `AbstractWindmillStream` so callers are just exposed to `Stream.create()`, `start()`, `halfClose()/shutdown()` as an interface and not have to worry about the restart mechanism?
another thing we can do if we don't want to `awaitTermination` on a thread, we can use a scheduledExecutorService to schedule a task that wakes up once every `0.75 * deadline_duration` and checks if we need to initiate a restart, although this could add some complexity compared to the approach where we just have a thread where we `awaitTermination() -> halfClose() -> shutdown`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2045699293
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java:
##########
@@ -72,6 +73,11 @@
public class GrpcWindmillStreamFactory implements StatusDataProvider {
private static final long DEFAULT_STREAM_RPC_DEADLINE_SECONDS = 300;
+
+ // 15 minutes less than DIRECT_PATH_STREAM_DEADLINE to allow for drains.
Review Comment:
I didn't see this defined anywhere. Do you mean the value set in `withDirectPathDeadline()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2824089859
Reminder, please take a look at this pr: @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023644717
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkBudget> getWorkBudget;
+
+ @GuardedBy("activeGetWorkStream")
Review Comment:
setting the budget on a stream can affect how many items/bytes we fetch in the header (and subsequent extension calls).
wanted to simplify the stream mechanics by preventing setBudget() from being called mid stream creation. That way either we start the stream first, and then call set budget which will update the internal stream budget and send an extension (possibly) OR we set the budget first, and start the stream w/ the new budget.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2746477272
R: @scwhittle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2913974540
@scwhittle revisiting after https://github.com/apache/beam/compare/master...scwhittle:beam:substream is merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2841783734
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @johnjcasey added as fallback since no labels match configuration
R: @Abacn for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2035672102
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -161,14 +190,15 @@ protected final synchronized boolean trySend(RequestT request)
try {
requestObserver.onNext(request);
return true;
- } catch (ResettableThrowingStreamObserver.StreamClosedException e) {
+ } catch (StreamClosedException e) {
// Stream was broken, requests may be retried when stream is reopened.
}
return false;
}
@Override
+ @SuppressWarnings("FutureReturnValueIgnored")
Review Comment:
remove (maybe obsolete anyway now it looks like). ignoring these has caused issues before
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
I think it might be cleaner if we kept track of the released observer in AbstractWindmillStream because then we could treat it's on completion differently than the active stream.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
+ // Create a new stream to flush any pending requests and restart the deadline. If the stream
+ // is closed or shutdown, don't restart the stream here since a restart has already been
+ // scheduled.
+ startStream();
Review Comment:
Ideally we wouldn't resend the pending things that were for the stream we are shutting down (unless that stream has errors shutting down and they need retrying).
We could change the pending stuff to be kept per-physical-stream instead of just a single set in the sub-implementations. Then when the stream finishes we could move back whatever needs retrying to a pending set for the next new stream to consume.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -103,13 +113,30 @@ protected AbstractWindmillStream(
StreamObserverFactory streamObserverFactory,
Set<AbstractWindmillStream<?, ?>> streamRegistry,
int logEveryNStreamFailures,
- String backendWorkerToken) {
+ String backendWorkerToken,
+ WindmillStreamTTL streamTTL) {
this.backendWorkerToken = backendWorkerToken;
this.executor =
Executors.newSingleThreadExecutor(
new ThreadFactoryBuilder()
.setDaemon(true)
- .setNameFormat(createThreadName(debugStreamType, backendWorkerToken))
+ .setNameFormat(
+ createThreadName(debugStreamType, backendWorkerToken, "WindmillStream"))
+ .build());
+ this.restartExecutor =
Review Comment:
we should avoid starting executor etc if we're not restarting the streams (ie if ttl is never). That will reduce scope of this change to just the direct path streams setting this.
And then separately after baking some we can remove the stream cache used in other cases to do the timeout and instead use this.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/ResettableThrowingStreamObserver.java:
##########
@@ -159,4 +180,76 @@ static final class StreamClosedException extends Exception {
super(s);
}
}
+
+ static final class InternalStreamTimeout extends Throwable {
+ private static final InternalStreamTimeout INSTANCE = new InternalStreamTimeout();
+
+ private InternalStreamTimeout() {}
+
+ static boolean isInternalTimeout(Throwable t) {
+ while (t != null) {
+ if (t == INSTANCE) {
+ return true;
+ }
+ t = t.getCause();
+ }
+ return false;
+ }
+ }
+
+ private final class AsyncStreamCloser {
+ private final BlockingQueue<StreamObserver<T>> streamsToClose;
+ private final ExecutorService streamCloserExecutor;
+
+ @GuardedBy("this")
+ private boolean started;
+
+ private AsyncStreamCloser() {
+ streamsToClose = new LinkedBlockingQueue<>();
+ streamCloserExecutor =
+ Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setNameFormat("StreamCloserThread-%d").build());
+ }
+
+ private synchronized void start() {
+ if (!started) {
+ streamCloserExecutor.execute(
+ () -> {
+ while (!isPoisoned()) {
+ try {
+ timeoutStream(streamsToClose.take());
+ } catch (InterruptedException e) {
+ // Drain streamsToClose to prevent any dangling StreamObservers.
+ streamsToClose.forEach(this::timeoutStream);
+ break;
+ }
+ }
+ });
+ started = true;
+ }
+ }
+
+ private void timeoutStream(StreamObserver<T> streamObserver) {
+ try {
+ streamObserver.onError(InternalStreamTimeout.INSTANCE);
Review Comment:
On timeout we should try to close cleanly instead of with an error. That will be better for monitoring if we can avoid errors in the common cases but also avoids wasting work (like reads that windmill is performing which responses we could consume instead of retrying the read).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-3148400807
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2035958287
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
from chat conversation
Opted to use `ResettableThrowingStreamObserver` to close the streams.
- A background thread in `AbstractWindmillStream` calls `AbstractWindmillStream.restart()` based on the passed in stream TTL.
- `AbstractWindmillStream.restart()` calls `ResettableThrowingStreamObserver.release()` to async close the stream and `AbstractWindmillStream.startStream()` to reset the `ResettableThrowingStreamObserver` and call `onNewStream()` to send headers and flush any pending messages
- `ResettableThrowingStreamObserver.release()` offloads the delegate to a queue which is polled by a background thread which calls `delegate.onError(InternalStreamTimeout())`
- `AbstractWindmillStream.ResponseObserver.onError(...)` (which is the delegate) treats `InternalStreamTimeout` differently than other errors, skipping the call the `executor.execute(AbstractWindmillStream.this::startStream)`, since we already restarted the stream in `AbstractWindmillStream.restart()`
now the `WindmillStreamSender` and callers can just call start() and shutdown() to manage the streams.
I only added the directpath changes in this PR, the older implementations still have the pattern of `start()/awaitTermination()/halfClose()` in cloudpath. The older code just submits a streamTTL w/ a negative time value which is skipped by `AbstractWindmillStream` when scheduling the background threads.
also added a way to retain the reference to the restart task via the future. I realized we don't want fixed intervals for stream restarts, we want restart time + the timeout, which changes every time we call startStream(). I think this should prevent a bunch of restarts from piling up and/or a restart to be called right after the stream has already restarted via other means.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2791877888
back to you @scwhittle thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073681134
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
got it so `AbstractWindmillStream` can have a 1 -> many relationship w/ the ResettableThrowingRequestObserver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2074207015
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
what would we want to do in this case
AbstractWindmillStream has an internal stream observer A
1. Internal Stream Observer A sends a request 1
2. Internal Stream Observer A times out and is released and scheduled for closure
3. AbstractWindmillStream schedules Internal StreamObserver A for closure and creates Internal StreamObserver B
4. Internal Stream Observer B starts (resend headers), and is ready for outbound/inbound requests/responses
5. a Response comes back to Internal Stream Observer A corresponding to request 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2074207015
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
what would we want to do in this case
AbstractWindmillStream
- internal stream observer A
- Internal Stream Observer A sends a request 1
- Internal Stream Observer A times out and is released and scheduled for closure
- AbstractWindmillStream schedules Internal StreamObserver A for closure and creates Internal StreamObserver B
- Internal Stream Observer B starts (resend headers), and is ready for outbound/inbound requests/responses
- a Response comes back to Internal Stream Observer A corresponding to request 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023650578
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
Review Comment:
done added a lock object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023695959
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java:
##########
@@ -189,6 +189,12 @@ private static <T extends AbstractStub<T>> T withDefaultDeadline(T stub) {
return stub.withDeadlineAfter(DEFAULT_STREAM_RPC_DEADLINE_SECONDS, TimeUnit.SECONDS);
}
+ private static <T extends AbstractStub<T>> T withLongDeadline(T stub) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2027001535
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
Did we discuss before adding this functionality directly to AbstractWindmillStream? It already has multiple physical grpc streams beneath it and knows how to restart things if needed. I think we'd have to make some of it's internal state per physical stream but the nice thing is that it would keep the clients simple as they just have a single logical stream.
Currently we have AbstractWindmillStream.halfClose() which means we want to terminate the logical stream and then we halfclose the physical stream (and new ones that we may create). I think that if we added some timeout/recreation internal to the abstractstream we could halfclose on physical stream that we want to teardown but the logical stream is not half-closed. We'd have to maintain a set of the background streams that were half-closed but not yet terminated.
I don't think the thread for management will cause CPU issues as noted since they are just WAITING, but since we can have a large # of windmill workers, I'd rather not just create a thread to block if not needed. We've had problems before where high # of threads use a lot of memory or perhaps slow down stack tracing etc. Some options would be some kind of scheduled future that we cancel if we observe a failure before then, or some maybeRestartStream method that a single thread at a higher level calls on all of the substreams like we do for maybeSendHeartbeats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025699431
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
how about adding an internal mechanism (maybe via another executor to make it async if we dont want the old stream closing to block the new stream creation/stream restart).
Where we add this workflow:
- create stream
- start stream
- wait for termination based on a TTL/timeout
- loop while(stream.isRunning).
we can add this in `AbstractWindmillStream` so callers are just exposed to `Stream.create()`, `start()`, `halfClose()/shutdown()` as an interface and not have to worry about the restart mechanism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025709752
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
awaitTermination() uses `CountDownLatch` underneath, so the thread would not be in a `BLOCKED` state I believe, just `TIMED_WAITING`, other threads can be scheduled in the mean time for other tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073738102
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/ResettableThrowingStreamObserver.java:
##########
@@ -159,4 +180,76 @@ static final class StreamClosedException extends Exception {
super(s);
}
}
+
+ static final class InternalStreamTimeout extends Throwable {
+ private static final InternalStreamTimeout INSTANCE = new InternalStreamTimeout();
+
+ private InternalStreamTimeout() {}
+
+ static boolean isInternalTimeout(Throwable t) {
+ while (t != null) {
+ if (t == INSTANCE) {
+ return true;
+ }
+ t = t.getCause();
+ }
+ return false;
+ }
+ }
+
+ private final class AsyncStreamCloser {
+ private final BlockingQueue<StreamObserver<T>> streamsToClose;
+ private final ExecutorService streamCloserExecutor;
+
+ @GuardedBy("this")
+ private boolean started;
+
+ private AsyncStreamCloser() {
+ streamsToClose = new LinkedBlockingQueue<>();
+ streamCloserExecutor =
+ Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setNameFormat("StreamCloserThread-%d").build());
+ }
+
+ private synchronized void start() {
+ if (!started) {
+ streamCloserExecutor.execute(
+ () -> {
+ while (!isPoisoned()) {
+ try {
+ timeoutStream(streamsToClose.take());
+ } catch (InterruptedException e) {
+ // Drain streamsToClose to prevent any dangling StreamObservers.
+ streamsToClose.forEach(this::timeoutStream);
+ break;
+ }
+ }
+ });
+ started = true;
+ }
+ }
+
+ private void timeoutStream(StreamObserver<T> streamObserver) {
+ try {
+ streamObserver.onError(InternalStreamTimeout.INSTANCE);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073836342
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -195,6 +228,12 @@ private void startStream() {
onNewStream();
if (clientClosed) {
halfClose();
+ } else {
+ // If the stream is alive, schedule a future restart. We also attempt to cancel any
+ // pending restarts that have not run since they are used for preventing
+ // DEADLINE_EXCEEDED. Everytime the stream restarts, this deadline restarts so we want
+ // the stream restart to run whenever we restart the stream + streamTTL.
+ scheduleRestart();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073821600
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
or perhaps release somehow releases some handle on the internal observer. We don't need the full feature set on these "released" streams. We want them half-closed, then we don't need to send anymore, we just need to be able to distinguish the responses coming on it, which requests were sent on it previously (in case it fails), and want to be able to abort it if it is taking too long to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2746477776
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025713645
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
Review Comment:
in the current case the new budget maybe reflected in the old and new stream which is ok, since the new stream will send the header + new budget in the request this seems like it could be simpler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2045680536
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -120,21 +147,23 @@ protected AbstractWindmillStream(
this.isHealthCheckScheduled = new AtomicBoolean(false);
this.finishLatch = new CountDownLatch(1);
this.logger = logger;
+ this.sleeper = Sleeper.DEFAULT;
+ this.debugMetrics = StreamDebugMetrics.create();
+ this.streamTTL = streamTTL;
this.requestObserver =
new ResettableThrowingStreamObserver<>(
() ->
streamObserverFactory.from(
clientFactory,
new AbstractWindmillStream<RequestT, ResponseT>.ResponseObserver()),
logger);
- this.sleeper = Sleeper.DEFAULT;
- this.debugMetrics = StreamDebugMetrics.create();
}
- private static String createThreadName(String streamType, String backendWorkerToken) {
+ private static String createThreadName(
+ String threadType, String streamType, String backendWorkerToken) {
Review Comment:
This doesn't quite match how we're using it. It looks like above we're passing in `(streamType, backendWorkerToken, threadType)`. It seems like we shouldn't need to hardcode "WindmillStream" in this method either, since it'll be passed in as threadType somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023650930
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
Review Comment:
can't lock on null since if we try to synchronize on null, i believe it will throw an NPE
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkBudget> getWorkBudget;
Review Comment:
done added a lock object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #34367: add graceful restart mechanism for GetWorkStream to prevent DEADLINE_…
URL: https://github.com/apache/beam/pull/34367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2741772940
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2749923185
Run Java precommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073833038
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
+ // Create a new stream to flush any pending requests and restart the deadline. If the stream
+ // is closed or shutdown, don't restart the stream here since a restart has already been
+ // scheduled.
+ startStream();
Review Comment:
opted to use a completely new stream instance after the timeout and let the underlying stream drain in the `AsyncStreamCloser`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2891014767
stop reviewer notifications
(keeping sticky with Sam/Andrew)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2748479514
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023652591
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -88,19 +105,24 @@ private WindmillStreamSender(
streamingEngineStreamFactory.createDirectCommitWorkStream(
connection, streamingEngineThrottleTimers.commitWorkThrottleTimer());
this.workCommitter = workCommitterFactory.apply(commitWorkStream);
- this.getWorkStream =
- streamingEngineStreamFactory.createDirectGetWorkStream(
- connection,
- withRequestBudget(getWorkRequest, getWorkBudget.get()),
- streamingEngineThrottleTimers.getWorkThrottleTimer(),
- FixedStreamHeartbeatSender.create(getDataStream),
- getDataClientFactory.apply(getDataStream),
- workCommitter,
- workItemScheduler);
+ this.activeGetWorkStream = new AtomicReference<>();
+ this.getWorkStreamFactory =
+ () ->
+ streamingEngineStreamFactory.createDirectGetWorkStream(
+ connection,
+ withRequestBudget(getWorkRequest, getWorkBudget.get()),
+ streamingEngineThrottleTimers.getWorkThrottleTimer(),
Review Comment:
`streamingEngineThrottleTimers.getWorkThrottleTimer()` returns the same throttle timer, will use the same for the others.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023639907
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023701614
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
+ }
+
+ // If graceful termination is unsuccessful, forcefully shutdown.
+ if (!newStream.awaitTermination(30, TimeUnit.SECONDS)) {
Review Comment:
bumped to 10x to 5 minutes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2024330828
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
Review Comment:
done offloaded closure to a different thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2876269357
Reminder, please take a look at this pr: @johnjcasey @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2748483846
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @johnjcasey added as fallback since no labels match configuration
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2016168525
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -88,19 +105,24 @@ private WindmillStreamSender(
streamingEngineStreamFactory.createDirectCommitWorkStream(
connection, streamingEngineThrottleTimers.commitWorkThrottleTimer());
this.workCommitter = workCommitterFactory.apply(commitWorkStream);
- this.getWorkStream =
- streamingEngineStreamFactory.createDirectGetWorkStream(
- connection,
- withRequestBudget(getWorkRequest, getWorkBudget.get()),
- streamingEngineThrottleTimers.getWorkThrottleTimer(),
- FixedStreamHeartbeatSender.create(getDataStream),
- getDataClientFactory.apply(getDataStream),
- workCommitter,
- workItemScheduler);
+ this.activeGetWorkStream = new AtomicReference<>();
+ this.getWorkStreamFactory =
+ () ->
+ streamingEngineStreamFactory.createDirectGetWorkStream(
+ connection,
+ withRequestBudget(getWorkRequest, getWorkBudget.get()),
+ streamingEngineThrottleTimers.getWorkThrottleTimer(),
+ FixedStreamHeartbeatSender.create(getDataStream),
+ getDataClientFactory.apply(getDataStream),
+ workCommitter,
+ workItemScheduler);
// 3 threads, 1 for each stream type (GetWork, GetData, CommitWork).
this.streamStarter =
Executors.newFixedThreadPool(
Review Comment:
how about newCachedThreadPool? it seems like 2 of these threads are just for start() and then won't be used and we might as well have them go away.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -88,19 +105,24 @@ private WindmillStreamSender(
streamingEngineStreamFactory.createDirectCommitWorkStream(
connection, streamingEngineThrottleTimers.commitWorkThrottleTimer());
this.workCommitter = workCommitterFactory.apply(commitWorkStream);
- this.getWorkStream =
- streamingEngineStreamFactory.createDirectGetWorkStream(
- connection,
- withRequestBudget(getWorkRequest, getWorkBudget.get()),
- streamingEngineThrottleTimers.getWorkThrottleTimer(),
- FixedStreamHeartbeatSender.create(getDataStream),
- getDataClientFactory.apply(getDataStream),
- workCommitter,
- workItemScheduler);
+ this.activeGetWorkStream = new AtomicReference<>();
+ this.getWorkStreamFactory =
+ () ->
+ streamingEngineStreamFactory.createDirectGetWorkStream(
+ connection,
+ withRequestBudget(getWorkRequest, getWorkBudget.get()),
+ streamingEngineThrottleTimers.getWorkThrottleTimer(),
Review Comment:
should we just use the same throttle timer, heartbeat sender, and getdataclientfactory for each stream?
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkBudget> getWorkBudget;
+
+ @GuardedBy("activeGetWorkStream")
Review Comment:
this doesn't seem like it needs to be guarded? any reason it has to be serially called?
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
Review Comment:
maybe this coudl be outside the synchronized block? we can create the new stream before swapping it in as the active one. We could create the new stream with no budget and just set it's budget once the old stream is half-closed.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -126,35 +148,49 @@ private static GetWorkRequest withRequestBudget(GetWorkRequest request, GetWorkB
}
synchronized void start() {
- if (!started.get()) {
+ if (isRunning.compareAndSet(false, true)) {
checkState(!streamStarter.isShutdown(), "WindmillStreamSender has already been shutdown.");
-
// Start these 3 streams in parallel since they each may perform blocking IO.
+ CountDownLatch waitForInitialStream = new CountDownLatch(1);
+ streamStarter.execute(() -> getWorkStreamLoop(waitForInitialStream));
CompletableFuture.allOf(
- CompletableFuture.runAsync(getWorkStream::start, streamStarter),
CompletableFuture.runAsync(getDataStream::start, streamStarter),
CompletableFuture.runAsync(commitWorkStream::start, streamStarter))
.join();
+ try {
+ waitForInitialStream.await();
+ } catch (InterruptedException e) {
+ close();
+ LOG.error("GetWorkStream to {} was never able to start.", backendWorkerToken);
+ throw new IllegalStateException("GetWorkStream unable to start aborting.", e);
+ }
workCommitter.start();
- started.set(true);
}
}
@Override
public synchronized void close() {
+ isRunning.set(false);
streamStarter.shutdownNow();
- getWorkStream.shutdown();
getDataStream.shutdown();
workCommitter.stop();
commitWorkStream.shutdown();
}
@Override
public void setBudget(long items, long bytes) {
- GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
- getWorkBudget.set(budget);
- if (started.get()) {
- getWorkStream.setBudget(budget);
+ synchronized (activeGetWorkStream) {
+ GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
Review Comment:
this build and setting on the atomic, could be outside synchronized block
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
Review Comment:
if this is guarded I don't think it needs to be an atomic.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -126,35 +148,49 @@ private static GetWorkRequest withRequestBudget(GetWorkRequest request, GetWorkB
}
synchronized void start() {
- if (!started.get()) {
+ if (isRunning.compareAndSet(false, true)) {
checkState(!streamStarter.isShutdown(), "WindmillStreamSender has already been shutdown.");
-
// Start these 3 streams in parallel since they each may perform blocking IO.
+ CountDownLatch waitForInitialStream = new CountDownLatch(1);
+ streamStarter.execute(() -> getWorkStreamLoop(waitForInitialStream));
CompletableFuture.allOf(
- CompletableFuture.runAsync(getWorkStream::start, streamStarter),
CompletableFuture.runAsync(getDataStream::start, streamStarter),
CompletableFuture.runAsync(commitWorkStream::start, streamStarter))
.join();
+ try {
+ waitForInitialStream.await();
+ } catch (InterruptedException e) {
+ close();
+ LOG.error("GetWorkStream to {} was never able to start.", backendWorkerToken);
+ throw new IllegalStateException("GetWorkStream unable to start aborting.", e);
+ }
workCommitter.start();
- started.set(true);
}
}
@Override
public synchronized void close() {
+ isRunning.set(false);
streamStarter.shutdownNow();
- getWorkStream.shutdown();
getDataStream.shutdown();
workCommitter.stop();
commitWorkStream.shutdown();
}
@Override
public void setBudget(long items, long bytes) {
- GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
- getWorkBudget.set(budget);
- if (started.get()) {
- getWorkStream.setBudget(budget);
+ synchronized (activeGetWorkStream) {
+ GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
+ getWorkBudget.set(budget);
+ if (isRunning.get()) {
Review Comment:
remove running check and just use the null below? seems like if activeGetWorkStream is set it is ok to call and it's one less interleaving to think about
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
private final ExecutorService streamStarter;
+ private final String backendWorkerToken;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkStream> activeGetWorkStream;
+
+ @GuardedBy("activeGetWorkStream")
+ private final AtomicReference<GetWorkBudget> getWorkBudget;
Review Comment:
ditto if this is guarded probably doesn't need to be atomic
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
+ }
+
+ // If graceful termination is unsuccessful, forcefully shutdown.
+ if (!newStream.awaitTermination(30, TimeUnit.SECONDS)) {
+ newStream.shutdown();
+ }
+
+ } catch (InterruptedException e) {
+ // continue until !isRunning.
Review Comment:
Could we instead force that isRunning is set to false here? we don't expect interruptions to happen for any other reason.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,15 +64,26 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
Review Comment:
could note that this needs to be less than the deadline in the other file
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
+ }
+
+ // If graceful termination is unsuccessful, forcefully shutdown.
+ if (!newStream.awaitTermination(30, TimeUnit.SECONDS)) {
Review Comment:
maybe better to increase this? If we lose getwork responses then windmill worker has to retry, if we can get them to flush with a little more time that seems fine.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java:
##########
@@ -189,6 +189,12 @@ private static <T extends AbstractStub<T>> T withDefaultDeadline(T stub) {
return stub.withDeadlineAfter(DEFAULT_STREAM_RPC_DEADLINE_SECONDS, TimeUnit.SECONDS);
}
+ private static <T extends AbstractStub<T>> T withLongDeadline(T stub) {
Review Comment:
withDirectPathDeadline?
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
+ newStream.start();
+ waitForInitialStream.countDown();
+ activeGetWorkStream.set(newStream);
+ }
+ try {
+ // Try to gracefully terminate the stream.
+ if (!newStream.awaitTermination(GET_WORK_STREAM_TTL_MINUTES, TimeUnit.MINUTES)) {
+ newStream.halfClose();
Review Comment:
I think as we half-close here we probably want to create a new stream to take over.
That way we aren't idle while we're waiting for the termination.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/observers/DirectStreamObserver.java:
##########
@@ -217,11 +217,11 @@ public void terminate(Throwable terminationException) {
}
private String constructStreamCancelledErrorMessage(long totalSecondsWaited) {
- return deadlineSeconds > 0
+ return inactivityTimeout > 0
? "Waited "
+ totalSecondsWaited
+ "s which exceeds given deadline of "
- + deadlineSeconds
+ + inactivityTimeout
+ "s for the outboundObserver to become ready meaning "
+ "that the stream deadline was not respected."
Review Comment:
this seems like the wrong message if it isn't the stream deadline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2023658059
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -126,35 +148,49 @@ private static GetWorkRequest withRequestBudget(GetWorkRequest request, GetWorkB
}
synchronized void start() {
- if (!started.get()) {
+ if (isRunning.compareAndSet(false, true)) {
checkState(!streamStarter.isShutdown(), "WindmillStreamSender has already been shutdown.");
-
// Start these 3 streams in parallel since they each may perform blocking IO.
+ CountDownLatch waitForInitialStream = new CountDownLatch(1);
+ streamStarter.execute(() -> getWorkStreamLoop(waitForInitialStream));
CompletableFuture.allOf(
- CompletableFuture.runAsync(getWorkStream::start, streamStarter),
CompletableFuture.runAsync(getDataStream::start, streamStarter),
CompletableFuture.runAsync(commitWorkStream::start, streamStarter))
.join();
+ try {
+ waitForInitialStream.await();
+ } catch (InterruptedException e) {
+ close();
+ LOG.error("GetWorkStream to {} was never able to start.", backendWorkerToken);
+ throw new IllegalStateException("GetWorkStream unable to start aborting.", e);
+ }
workCommitter.start();
- started.set(true);
}
}
@Override
public synchronized void close() {
+ isRunning.set(false);
streamStarter.shutdownNow();
- getWorkStream.shutdown();
getDataStream.shutdown();
workCommitter.stop();
commitWorkStream.shutdown();
}
@Override
public void setBudget(long items, long bytes) {
- GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
- getWorkBudget.set(budget);
- if (started.get()) {
- getWorkStream.setBudget(budget);
+ synchronized (activeGetWorkStream) {
+ GetWorkBudget budget = GetWorkBudget.builder().setItems(items).setBytes(bytes).build();
+ getWorkBudget.set(budget);
+ if (isRunning.get()) {
Review Comment:
setBudget on the `GetWorkStream` object may try to send an extension, should we make the `GetWorkStream` check isStarted before doing that? or this would prevent that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025714170
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -165,4 +201,39 @@ long getAndResetThrottleTime() {
long getCurrentActiveCommitBytes() {
return workCommitter.currentActiveCommitBytes();
}
+
+ /**
+ * Creates, starts, and gracefully terminates {@link GetWorkStream} before the clientside deadline
+ * to prevent {@link org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status#DEADLINE_EXCEEDED} errors.
+ * If at any point the server closes the stream, reconnects immediately.
+ */
+ private void getWorkStreamLoop(CountDownLatch waitForInitialStream) {
+ @Nullable GetWorkStream newStream = null;
+ while (isRunning.get()) {
+ synchronized (activeGetWorkStream) {
+ newStream = getWorkStreamFactory.get();
Review Comment:
allows the new `GetWorkBudget` to be propogated without coordinating any further sends
wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2025120871
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSender.java:
##########
@@ -57,28 +64,42 @@
@Internal
@ThreadSafe
final class WindmillStreamSender implements GetWorkBudgetSpender, StreamSender {
- private static final String STREAM_STARTER_THREAD_NAME = "StartWindmillStreamThread-%d";
- private final AtomicBoolean started;
- private final AtomicReference<GetWorkBudget> getWorkBudget;
- private final GetWorkStream getWorkStream;
+ private static final Logger LOG = LoggerFactory.getLogger(WindmillStreamSender.class);
+ private static final String STREAM_MANAGER_THREAD_NAME_FORMAT = "WindmillStreamManagerThread";
+
+ // Must be shorter than withLongDeadline() duration in GrpcWindmillStreamFactory to prevent
+ // DEADLINE_EXCEEDED.
+ private static final int GET_WORK_STREAM_TTL_MINUTES = 45;
+
+ private final AtomicBoolean isRunning = new AtomicBoolean(false);
private final GetDataStream getDataStream;
private final CommitWorkStream commitWorkStream;
private final WorkCommitter workCommitter;
private final StreamingEngineThrottleTimers streamingEngineThrottleTimers;
- private final ExecutorService streamStarter;
+ private final ExecutorService streamManagerExecutor;
+ private final String backendWorkerToken;
+ private final Object getWorkStreamLock = new Object();
+
+ @GuardedBy("getWorkStreamLock")
+ private final Supplier<GetWorkStream> getWorkStreamFactory;
+
+ @GuardedBy("getWorkStreamLock")
+ private @Nullable GetWorkStream activeGetWorkStream;
+
+ @GuardedBy("getWorkStreamLock")
+ private GetWorkBudget getWorkBudget;
private WindmillStreamSender(
WindmillConnection connection,
GetWorkRequest getWorkRequest,
- AtomicReference<GetWorkBudget> getWorkBudget,
+ GetWorkBudget initialGetWorkBudget,
GrpcWindmillStreamFactory streamingEngineStreamFactory,
WorkItemScheduler workItemScheduler,
Function<GetDataStream, GetDataClient> getDataClientFactory,
Function<CommitWorkStream, WorkCommitter> workCommitterFactory) {
- this.started = new AtomicBoolean(false);
- this.getWorkBudget = getWorkBudget;
+ this.backendWorkerToken = connection.backendWorkerToken();
+ this.getWorkBudget = initialGetWorkBudget;
this.streamingEngineThrottleTimers = StreamingEngineThrottleTimers.create();
-
// Stream instances connect/reconnect internally, so we can reuse the same instance through the
Review Comment:
don't we have the same problem that the GetData and commitWorkStream will have deadline exceeds?
I'm wondering if it would be better to either improve StreamPool or have a more reusable class like StreamPool for internally reconnecting an AbstractWindmillStream. It woudl also be nice if we didn't have to have an additional thread-per stream just waiting for it terminate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073833637
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java:
##########
@@ -72,6 +73,11 @@
public class GrpcWindmillStreamFactory implements StatusDataProvider {
private static final long DEFAULT_STREAM_RPC_DEADLINE_SECONDS = 300;
+
+ // 15 minutes less than DIRECT_PATH_STREAM_DEADLINE to allow for drains.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2074208905
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -333,6 +372,49 @@ public final synchronized void halfClose() {
}
}
+ /**
+ * Internally restart the stream to avoid {@link Status#DEADLINE_EXCEEDED} errors.
+ *
+ * @implNote Similar behavior to {@link #halfClose()}, except we allow callers to interact with
+ * the stream after restarts.
+ */
+ private synchronized void restart() throws WindmillStreamShutdownException {
+ debugMetrics.recordRestartReason("Internal Timeout");
+ try {
+ requestObserver.release();
Review Comment:
Do we want to add a flag to ResponseObserver marking it as timed out and handle the `onNext`, `onError`, and `onComplete` differently in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on code in PR #34367:
URL: https://github.com/apache/beam/pull/34367#discussion_r2073679778
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/AbstractWindmillStream.java:
##########
@@ -103,13 +113,30 @@ protected AbstractWindmillStream(
StreamObserverFactory streamObserverFactory,
Set<AbstractWindmillStream<?, ?>> streamRegistry,
int logEveryNStreamFailures,
- String backendWorkerToken) {
+ String backendWorkerToken,
+ WindmillStreamTTL streamTTL) {
this.backendWorkerToken = backendWorkerToken;
this.executor =
Executors.newSingleThreadExecutor(
new ThreadFactoryBuilder()
.setDaemon(true)
- .setNameFormat(createThreadName(debugStreamType, backendWorkerToken))
+ .setNameFormat(
+ createThreadName(debugStreamType, backendWorkerToken, "WindmillStream"))
+ .build());
+ this.restartExecutor =
Review Comment:
we only schedule the future if the ttl is valid in `scheduleRestart()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add graceful restart mechanism for GetWorkStream to prevent DEADLINE_… [beam]
Posted by "m-trieu (via GitHub)" <gi...@apache.org>.
m-trieu commented on PR #34367:
URL: https://github.com/apache/beam/pull/34367#issuecomment-2755988053
test failure is unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org