You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "LEEKYE (via GitHub)" <gi...@apache.org> on 2026/02/23 21:52:22 UTC
[PR] Omit state commits in stateful Dofn [beam]
LEEKYE opened a new pull request, #37691:
URL: https://github.com/apache/beam/pull/37691
- Addresses #37690
- Propagate 'has_no_state' and 'only_bundle_for_keys' of ProcessBundleRequest from ProcessBundleHandler to where StateRequest (StateAppendRequest, StateGetRequest) is constructed.
- If 'has_no_state' is true, StateGetRequest is skipped
- If 'only_bundle_for_keys' is true, StateAppendRequest is skipped.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950241828
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "ryan-mbuashundip (via GitHub)" <gi...@apache.org>.
ryan-mbuashundip commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2880690642
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -677,6 +683,15 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient,
+ StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = hasNoState ? null : stateRequestForFirstChunk.getGet().getContinuationToken();
Review Comment:
@Abacn Do you know what can be checked in order to verify that there is, in fact, state to be read from the runner without requesting that state?
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
Review Comment:
Calls to `seekToContinuationToken` might override the null `continuationToken` set in the iterator constructor, so you'll need to save the bit in the class and check it here to make sure the state request is never made.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -54,27 +54,33 @@ public class BagUserState<T> {
private List<T> newValues;
private boolean isCleared;
private boolean isClosed;
+ private final boolean hasNoState;
+ private final boolean onlyBundleForKeys;
static final int BAG_APPEND_BATCHING_LIMIT = 10 * 1024 * 1024;
/** The cache must be namespaced for this state object accordingly. */
public BagUserState(
- Cache<?, ?> cache,
- BeamFnStateClient beamFnStateClient,
- String instructionId,
- StateKey stateKey,
- Coder<T> valueCoder) {
+ Cache<?, ?> cache,
+ BeamFnStateClient beamFnStateClient,
+ String instructionId,
+ StateKey stateKey,
+ Coder<T> valueCoder,
+ boolean hasNoState,
+ boolean onlyBundleForKeys) {
Review Comment:
I think the indentation got messed up here.
```suggestion
Cache<?, ?> cache,
BeamFnStateClient beamFnStateClient,
String instructionId,
StateKey stateKey,
Coder<T> valueCoder,
boolean hasNoState,
boolean onlyBundleForKeys) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #37691: Omit state commits in stateful Dofn
URL: https://github.com/apache/beam/pull/37691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933822499
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java:
##########
@@ -114,6 +114,8 @@ public static class Factory<K> {
private final Map<TupleTag<?>, SideInputSpec> sideInputSpecMap;
private final Coder<K> keyCoder;
private final Coder<BoundedWindow> windowCoder;
+ private final boolean hasNoState;
+ private final boolean onlyBundleForKeys;
Review Comment:
I still don't think these can be booleans, but rather need to be `Supplier<boolean>`. For batch, it might be true that both values will be true for all work items and all keys, but for streaming that's not the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881170406
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
Review Comment:
If `onlyBundleForKeys` is true, can we just return here? We shouldn't need to send anything back across the FnApi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4138661260
before merging, would you mind running a beam_PostCommit_Java_ValidatesRunner_Dataflow_V2 test? That is making any change to https://github.com/apache/beam/blob/master/.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_V2.json and push the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4183263896
Reminder, please take a look at this pr: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2978275364
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java:
##########
@@ -235,7 +243,9 @@ public FnApiStateAccessor<K> create() {
sideInputSpecMap,
beamFnStateClient,
keyCoder,
- windowCoder);
+ windowCoder,
+ hasNoState.get(),
Review Comment:
After doing the above-mentioned test, we might find that these need to pass the Supplier to the FnApiStateAccessor as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881216203
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java:
##########
@@ -75,12 +77,13 @@ public MultimapSideInput(
this.keyCoder = keyCoder;
this.valueCoder = valueCoder;
this.useBulkRead = useBulkRead;
+ this.hasNoState = hasNoState;
}
@Override
public Iterable<K> get() {
return StateFetchingIterators.readAllAndDecodeStartingFrom(
Review Comment:
How does batch get its side inputs? We might still need to use the StateFetchingIterators here? Otherwise, with what you have now, I think `get()` will just always return an empty Iterable when `hasNoState` is true.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
As an alternative to consider: instead of passing `hasNoState` into the `StateFetchingIterators` class, what if we just set `this.persistedEntries` to some kind of "empty" PrefetchableIterator when `hasNoState` is true instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3947654287
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4360039773
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "ryan-mbuashundip (via GitHub)" <gi...@apache.org>.
ryan-mbuashundip commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933475222
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java:
##########
@@ -75,12 +77,13 @@ public MultimapSideInput(
this.keyCoder = keyCoder;
this.valueCoder = valueCoder;
this.useBulkRead = useBulkRead;
+ this.hasNoState = hasNoState;
}
@Override
public Iterable<K> get() {
return StateFetchingIterators.readAllAndDecodeStartingFrom(
Review Comment:
Good question, @Abacn or @kennknowles do you know if we need to use `StateFetchingIterators` for side inputs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "ryan-mbuashundip (via GitHub)" <gi...@apache.org>.
ryan-mbuashundip commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933475222
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java:
##########
@@ -75,12 +77,13 @@ public MultimapSideInput(
this.keyCoder = keyCoder;
this.valueCoder = valueCoder;
this.useBulkRead = useBulkRead;
+ this.hasNoState = hasNoState;
}
@Override
public Iterable<K> get() {
return StateFetchingIterators.readAllAndDecodeStartingFrom(
Review Comment:
Good question, @Abacn or @kennknowles do you know if we use `StateFetchingIterators` for side inputs in batch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933852846
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/IterableSideInput.java:
##########
@@ -40,17 +41,23 @@ public IterableSideInput(
BeamFnStateClient beamFnStateClient,
String instructionId,
StateKey stateKey,
- Coder<T> valueCoder) {
+ Coder<T> valueCoder,
+ boolean hasNoState) {
checkArgument(
stateKey.hasIterableSideInput(),
"Expected IterableSideInput StateKey but received %s.",
stateKey);
this.values =
- StateFetchingIterators.readAllAndDecodeStartingFrom(
- cache,
- beamFnStateClient,
- StateRequest.newBuilder().setInstructionId(instructionId).setStateKey(stateKey).build(),
- valueCoder);
+ hasNoState
Review Comment:
It surprises me that this works. Doesn't this mean for batch pipelines trying to call `get()` on a side input will always return an empty set? It looks like in the IterableSideINputTest.java below, we're setting `hasNoState` to false, so that's probably why the tests are passing. But in FnApiStateAccessor, we're potentially setting it to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2963151976
##########
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapUserStateTest.java:
##########
Review Comment:
We still need some tests where hasNoState and/or onlyBundleForKeys are true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6317880576
Pending internal (Dataflow portable runner integration) test result with this PR imported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6152531058
@Abacn Please reopen this pr for me thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881144965
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PTransformRunnerFactory.java:
##########
@@ -160,6 +160,18 @@ <T> void addIncomingTimerEndpoint(
* instant provides the timeout on how long the finalization callback is valid for.
*/
DoFn.BundleFinalizer getBundleFinalizer();
+
+ /**
+ * Returns true if the runner has no state for the keys in the ProcessBundleRequest. If true,
+ * the SDK can begin stateful processing with an initial empty state.
+ */
+ boolean getHasNoState();
Review Comment:
Note, it's probably ok to just pass `booleans` into the `*State` objects though since those are created per key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881201295
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
Ditto on `|| hasNoState` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881242079
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/IterableSideInput.java:
##########
@@ -50,7 +51,8 @@ public IterableSideInput(
cache,
beamFnStateClient,
StateRequest.newBuilder().setInstructionId(instructionId).setStateKey(stateKey).build(),
- valueCoder);
+ valueCoder,
Review Comment:
Ditto my question here about side inputs. We might still need to look them up over the FnApi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950247531
Assigning reviewers:
R: @Abacn for label java.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2885349795
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java:
##########
@@ -246,7 +246,9 @@ private void addRunnerAndConsumersForPTransformRecursively(
BundleFinalizer bundleFinalizer,
Collection<BeamFnDataReadRunner<?>> channelRoots,
Map<ApiServiceDescriptor, BeamFnDataOutboundAggregator> outboundAggregatorMap,
- Set<String> runnerCapabilities)
+ Set<String> runnerCapabilities,
+ boolean hasNoState,
Review Comment:
At least from my cl in uw side, when it is batch, both fields will be set to true, but I am not really sure if these two fields should have different values in different scenarios. Since these 2 are both new fields that have no occurrences anywhere else, I don't think they will have different values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6315968018
Reminder, please take a look at this pr: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2903152657
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
DONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2843507744
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -669,12 +675,23 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
private final StateRequest stateRequestForFirstChunk;
private ByteString continuationToken;
private CompletableFuture<StateResponse> prefetchedResponse;
+ private boolean hasNoState;
LazyBlockingStateFetchingIterator(
BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk) {
this.beamFnStateClient = beamFnStateClient;
this.stateRequestForFirstChunk = stateRequestForFirstChunk;
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
+ this.hasNoState = false;
+ }
+
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
+ this.hasNoState = hasNoState;
}
Review Comment:

There's a potential `NullPointerException` here. If `hasNoState` is true, `continuationToken` can still be non-null (e.g., `ByteString.EMPTY` for a new read). This would cause `hasNext()` to return `true`. However, `prefetch()` will do nothing due to the `!hasNoState` check. Consequently, when `next()` is called, it will attempt to call `.get()` on a `null` `prefetchedResponse`, leading to an NPE.
To fix this, the iterator should be made empty when `hasNoState` is true. The cleanest way to achieve this is by setting `continuationToken` to `null` in the constructor if `hasNoState` is true. This will cause `hasNext()` to return `false` from the beginning.
```java
LazyBlockingStateFetchingIterator(
BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk,
boolean hasNoState) {
this.beamFnStateClient = beamFnStateClient;
this.stateRequestForFirstChunk = stateRequestForFirstChunk;
this.hasNoState = hasNoState;
if (hasNoState) {
this.continuationToken = null;
} else {
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
}
```
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -713,7 +730,7 @@ public boolean isReady() {
@Override
public void prefetch() {
- if (continuationToken != null && prefetchedResponse == null) {
+ if (continuationToken != null && prefetchedResponse == null && !hasNoState) {
Review Comment:

If the constructor is updated to nullify `continuationToken` when `hasNoState` is true (as suggested in another comment), this `!hasNoState` check becomes redundant because `continuationToken != null` will already be false in that case. You could simplify this condition for better readability.
```java
if (continuationToken != null && prefetchedResponse == null) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4925551022
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881249699
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java:
##########
Review Comment:
I assume we'll need to plumb `hasNoState` and `onlyBundleForKey` into all of these `create*UserState` methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881253783
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java:
##########
Review Comment:
Oh, I see. We just bypassed the function arguments and pass them into the constructor inside the create methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2885470096
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java:
##########
@@ -246,7 +246,9 @@ private void addRunnerAndConsumersForPTransformRecursively(
BundleFinalizer bundleFinalizer,
Collection<BeamFnDataReadRunner<?>> channelRoots,
Map<ApiServiceDescriptor, BeamFnDataOutboundAggregator> outboundAggregatorMap,
- Set<String> runnerCapabilities)
+ Set<String> runnerCapabilities,
+ boolean hasNoState,
Review Comment:
For streaming jobs I suspect we'll always have `onlyBundleForKeys = false`, but the underlying execution engine can give hints about whether or not `hasNoState` is true or false. But I can't think of an example where we'd have them set in my above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2843409460
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -669,12 +675,23 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
private final StateRequest stateRequestForFirstChunk;
private ByteString continuationToken;
private CompletableFuture<StateResponse> prefetchedResponse;
+ private boolean hasNoState;
LazyBlockingStateFetchingIterator(
BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk) {
this.beamFnStateClient = beamFnStateClient;
this.stateRequestForFirstChunk = stateRequestForFirstChunk;
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
+ this.hasNoState = false;
+ }
+
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
+ this.hasNoState = hasNoState;
}
Review Comment:

If `hasNoState` is true, the `continuationToken` should be initialized to `null`. This ensures that `hasNext()` returns `false` and prevents `next()` from being called. Currently, if `hasNoState` is true, `continuationToken` is still initialized from the request, `hasNext()` returns `true`, and `next()` eventually throws a `NullPointerException` because `prefetchedResponse` is never initialized in `prefetch()`.
```suggestion
LazyBlockingStateFetchingIterator(
BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk,
boolean hasNoState) {
this.beamFnStateClient = beamFnStateClient;
this.stateRequestForFirstChunk = stateRequestForFirstChunk;
this.continuationToken =
hasNoState ? null : stateRequestForFirstChunk.getGet().getContinuationToken();
this.hasNoState = hasNoState;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2845463297
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
@@ -438,7 +445,7 @@ private void startStateApiWrites() {
}
// Persist pending key-values
- if (!pendingAdds.isEmpty()) {
+ if (!pendingAdds.isEmpty() && !onlyBundleForKeys) {
Review Comment:

The `StateClearRequest`s sent when `isCleared` is true or `pendingRemoves` is not empty are not guarded by `!onlyBundleForKeys`. To be consistent with skipping appends, should these clear operations also be skipped when `onlyBundleForKeys` is true?
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java:
##########
@@ -303,6 +309,11 @@ public void asyncClose() throws Exception {
pendingRemoves.clear();
}
+ if (onlyBundleForKeys) {
+ pendingAdds.clear();
+ return;
+ }
Review Comment:

Similar to my other comments, the logic for `pendingRemoves` which results in `StateClearRequest`s is not guarded by `onlyBundleForKeys`. To be consistent, we should probably skip sending those requests as well when `onlyBundleForKeys` is true. The current implementation only skips `pendingAdds`.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -135,7 +141,7 @@ public void asyncClose() throws Exception {
for (T newValue : newValues) {
int previousSize = out.size();
valueCoder.encode(newValue, out);
- if (out.size() > BAG_APPEND_BATCHING_LIMIT && previousSize > 0) {
+ if (out.size() > BAG_APPEND_BATCHING_LIMIT && previousSize > 0 && !onlyBundleForKeys) {
Review Comment:

There's a small performance improvement opportunity here. When `onlyBundleForKeys` is true, the code still iterates over `newValues` and encodes them, even though they are never sent to the runner.
To avoid this unnecessary work, you could wrap the entire block that processes `newValues` in an `if (!onlyBundleForKeys)` check. This is similar to the pattern used in `OrderedListUserState.java`.
For example:
```java
if (!newValues.isEmpty() && !onlyBundleForKeys) {
// Batch values up to a arbitrary limit ...
ByteStringOutputStream out = new ByteStringOutputStream();
for (T newValue : newValues) {
// ... existing encoding and sending logic ...
}
if (out.size() > 0) {
// ... existing sending logic ...
}
}
```
This would prevent the encoding from happening when the data isn't going to be sent.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -135,7 +141,7 @@ public void asyncClose() throws Exception {
for (T newValue : newValues) {
int previousSize = out.size();
valueCoder.encode(newValue, out);
- if (out.size() > BAG_APPEND_BATCHING_LIMIT && previousSize > 0) {
+ if (out.size() > BAG_APPEND_BATCHING_LIMIT && previousSize > 0 && !onlyBundleForKeys) {
Review Comment:

While you've correctly guarded the `StateAppendRequest`s with `!onlyBundleForKeys`, it seems the `StateClearRequest` is not. If `onlyBundleForKeys` is true, we should probably avoid sending any state modifications to the runner, including clears.
The `clear` operation at line 133 seems to be missing a check for `onlyBundleForKeys`:
```java
if (isCleared) { // This should probably be if (isCleared && !onlyBundleForKeys)
beamFnStateClient.handle(
request.toBuilder().setClear(StateClearRequest.getDefaultInstance()));
}
```
This would make the behavior consistent with not sending appends.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950247937
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2903068339
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
Review Comment:
I took the advice from Andrew that when calling StateFetchingIterators::readAllAndDecodeStartingFrom(), if hasNoState is true, returns emptyIterables, otherwise, call this function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933910058
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -94,6 +94,20 @@ public static <T> CachingStateIterable<T> readAllAndDecodeStartingFrom(
valueCoder);
}
+ public static <T> CachingStateIterable<T> emptyCachingStateIterable(
Review Comment:
This could probably use some testing. My concern is that, because we're using a noop cache, if we ever decide to check if such an iterable is empty, it'll do that by trying to read the first page of the iterable, which will then try to send a request across the FnApi (which shouldn't ever be needed). We might need to instead make some sort of subclass that always returns an empty iterator from its GetIterator method. At the very least, we shouldn't need to pass in a `BeamFnStateClient`, `StateRequest`, or even valueCoder I guess to such a thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933928045
##########
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/OrderedListUserStateTest.java:
##########
@@ -650,6 +689,75 @@ public void testOperationsDuringNavigatingIterable() throws Exception {
assertEquals(iter.next(), A4);
}
+ @Test
+ public void testHasNoState() throws Exception {
Review Comment:
Can we also add a test for readRange?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881136994
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PTransformRunnerFactory.java:
##########
@@ -160,6 +160,18 @@ <T> void addIncomingTimerEndpoint(
* instant provides the timeout on how long the finalization callback is valid for.
*/
DoFn.BundleFinalizer getBundleFinalizer();
+
+ /**
+ * Returns true if the runner has no state for the keys in the ProcessBundleRequest. If true,
+ * the SDK can begin stateful processing with an initial empty state.
+ */
+ boolean getHasNoState();
Review Comment:
In all of this plumbing, I'm wondering if these should be `Supplier<boolean>`'s instead of booleans. We reuse a bunch of these objects across many different bundles and, in theory, different bundles could have different values for `hasNoState` and `onlyBundleForKeys` (though I think for batch it'll just always be true).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933852846
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/IterableSideInput.java:
##########
@@ -40,17 +41,23 @@ public IterableSideInput(
BeamFnStateClient beamFnStateClient,
String instructionId,
StateKey stateKey,
- Coder<T> valueCoder) {
+ Coder<T> valueCoder,
+ boolean hasNoState) {
checkArgument(
stateKey.hasIterableSideInput(),
"Expected IterableSideInput StateKey but received %s.",
stateKey);
this.values =
- StateFetchingIterators.readAllAndDecodeStartingFrom(
- cache,
- beamFnStateClient,
- StateRequest.newBuilder().setInstructionId(instructionId).setStateKey(stateKey).build(),
- valueCoder);
+ hasNoState
Review Comment:
It surprises me that this works. Doesn't this mean for batch pipelines trying to call `get()` on a side input will always return an empty set? It looks like in the IterableSideInputTest.java below, we're setting `hasNoState` to false, so that's probably why the tests are passing. But in FnApiStateAccessor, we're potentially setting it to true. Maybe we need to beef up our testing of FnApiStateAccessor to test something like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933856475
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java:
##########
@@ -75,16 +78,23 @@ public MultimapSideInput(
this.keyCoder = keyCoder;
this.valueCoder = valueCoder;
this.useBulkRead = useBulkRead;
+ this.hasNoState = hasNoState;
}
@Override
public Iterable<K> get() {
- return StateFetchingIterators.readAllAndDecodeStartingFrom(
- cache, beamFnStateClient, keysRequest, keyCoder);
+ return this.hasNoState
+ ? PrefetchableIterables.emptyIterable()
Review Comment:
Ditto my concerns for this type of side input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2933923265
##########
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java:
##########
@@ -96,7 +97,8 @@ public void testGet() throws Exception {
keysStateKey(),
ByteArrayCoder.of(),
StringUtf8Coder.of(),
- true);
+ true,
+ false);
Review Comment:
Need some tests with set to true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2957926562
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -94,6 +94,20 @@ public static <T> CachingStateIterable<T> readAllAndDecodeStartingFrom(
valueCoder);
}
+ public static <T> CachingStateIterable<T> emptyCachingStateIterable(
Review Comment:
I made a change to return emptyIterator for EMPTY_ITERABLE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4183680396
We found some runner v2 internal tests failing and currently resolving it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2843278955
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java:
##########
@@ -161,7 +162,9 @@ public OrderedListUserState(
BeamFnStateClient beamFnStateClient,
String instructionId,
StateKey stateKey,
- Coder<T> valueCoder) {
+ Coder<T> valueCoder,
+ boolean hasNoState,
+ boolean onlyBundleForKeys) {
Review Comment:

The `onlyBundleForKeys` parameter is stored but appears to be unused in `asyncClose` to skip `StateAppendRequest`s. This seems to be an omission, as `BagUserState` uses this flag to optimize writes. Please ensure this flag is used to skip appending state when it's true.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -328,16 +330,19 @@ public static <T> Block<T> fromValues(
private final BeamFnStateClient beamFnStateClient;
private final StateRequest stateRequestForFirstChunk;
private final Coder<T> valueCoder;
+ private final boolean hasNoState;
public CachingStateIterable(
Cache<IterableCacheKey, Blocks<T>> cache,
BeamFnStateClient beamFnStateClient,
StateRequest stateRequestForFirstChunk,
- Coder<T> valueCoder) {
+ Coder<T> valueCoder,
+ boolean hasNoState) {
this.cache = cache;
this.beamFnStateClient = beamFnStateClient;
this.stateRequestForFirstChunk = stateRequestForFirstChunk;
this.valueCoder = valueCoder;
+ this.hasNoState = hasNoState;
Review Comment:

The `hasNoState` field is correctly stored here, but it seems it's not being passed to the `LazyBlockingStateFetchingIterator` within the inner class `CachingStateIterator`. The `CachingStateIterator` constructor should be updated to use the new `LazyBlockingStateFetchingIterator` constructor that accepts `hasNoState`. Without this change, state fetches won't be skipped even when `hasNoState` is true.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
@@ -84,7 +86,9 @@ public MultimapUserState(
String instructionId,
StateKey stateKey,
Coder<K> mapKeyCoder,
- Coder<V> valueCoder) {
+ Coder<V> valueCoder,
+ boolean hasNoState,
+ boolean onlyBundleForKeys) {
Review Comment:

The `onlyBundleForKeys` parameter is stored but appears to be unused in `asyncClose` to skip `StateAppendRequest`s. This seems to be an omission, as `BagUserState` uses this flag to optimize writes. Please ensure this flag is used to skip appending state when it's true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950186709
R: @apanich
R: @ryan-mbuashundip
R: @acrites
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881205474
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
Same as for other state types, I think we can return early after setting `isClosed` to true if `onlyBundleForKey` is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881200236
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
I think this could be `isCleared || hasNoState` since in either case, we can just return local additions only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2880569618
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -677,6 +683,15 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient,
+ StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = hasNoState ? null : stateRequestForFirstChunk.getGet().getContinuationToken();
Review Comment:
I am not super familiar with this, but if I understand correctly, 'hasNoState' indicates whether StateGetRequest will be constructed and processed, or just skipped to improve performance. And if StateGetRequest is processed to bring in the actual states, it does not make sense to plumb the 'hasNoState' field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2912028963
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
Review Comment:
I created a new method emptyCachingStateIterable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2845541378
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -54,27 +54,33 @@ public class BagUserState<T> {
private List<T> newValues;
private boolean isCleared;
private boolean isClosed;
+ private final boolean hasNoState;
+ private final boolean onlyBundleForKeys;
static final int BAG_APPEND_BATCHING_LIMIT = 10 * 1024 * 1024;
/** The cache must be namespaced for this state object accordingly. */
public BagUserState(
- Cache<?, ?> cache,
- BeamFnStateClient beamFnStateClient,
- String instructionId,
- StateKey stateKey,
- Coder<T> valueCoder) {
+ Cache<?, ?> cache,
+ BeamFnStateClient beamFnStateClient,
+ String instructionId,
+ StateKey stateKey,
+ Coder<T> valueCoder,
+ boolean hasNoState,
+ boolean onlyBundleForKeys) {
Review Comment:

The indentation for the constructor parameters has been changed from 4 spaces to 2 spaces. This is inconsistent with standard Java formatting guidelines (e.g., Google Java Style Guide recommends an indent of +4 for continuation lines) and the previous version of the code. Please restore the 4-space indentation to improve readability.
```suggestion
public BagUserState(
Cache<?, ?> cache,
BeamFnStateClient beamFnStateClient,
String instructionId,
StateKey stateKey,
Coder<T> valueCoder,
boolean hasNoState,
boolean onlyBundleForKeys) {
```
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -677,6 +683,14 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient, StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
Review Comment:

The formatting of the new `LazyBlockingStateFetchingIterator` constructor is inconsistent. To improve readability and adhere to common Java style conventions (like the Google Java Style Guide), it's better to place each parameter on a new line with consistent indentation, especially since they don't fit on a single line.
```suggestion
LazyBlockingStateFetchingIterator(
BeamFnStateClient beamFnStateClient,
StateRequest stateRequestForFirstChunk,
boolean hasNoState) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950192249
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3947530173
## Summary of Changes
Hello @LEEKYE, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enhances the Apache Beam Fn Harness by introducing mechanisms to optimize state management for stateful DoFns. It allows the runner to signal whether a bundle has no initial state or if it's the final bundle for specific keys, meaning no further state needs to be committed. By propagating these flags, the system can intelligently skip unnecessary state read and write operations, improving efficiency for certain stateful processing scenarios.
### Highlights
* **State Optimization Flags**: Introduced `hasNoState` and `onlyBundleForKeys` flags to optimize stateful DoFn processing within the Fn Harness.
* **Conditional State Reads**: Implemented logic to skip `StateGetRequest` operations when `hasNoState` is true, preventing unnecessary state fetches.
* **Conditional State Writes**: Added conditions to prevent `StateAppendRequest` operations when `onlyBundleForKeys` is true, avoiding state commits for completed key processing.
* **Flag Propagation**: Propagated these state optimization flags through the `ProcessBundleHandler` and various state accessor and user state classes to ensure consistent behavior.
🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2964207738
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
@@ -93,14 +98,18 @@ public MultimapUserState(
this.beamFnStateClient = beamFnStateClient;
this.mapKeyCoder = mapKeyCoder;
this.valueCoder = valueCoder;
+ this.hasNoState = hasNoState;
+ this.onlyBundleForKeys = onlyBundleForKeys;
// Note: These StateRequest protos are constructed even if we never try to read the
// corresponding state type. Consider constructing them lazily, as needed.
Review Comment:
There is a comment "Note: These StateRequest protos are constructed even if we never try to read corresponding state type. Consider constructing them lazily, as needed." So I believe it might be necessary to construct the state requests for some reason even though they will not be used when hasNoState or onlyBundleForKeys is true. Right now, the staterequests will not be sent when hasNoState or onlyBundleForKeys is true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2958007273
##########
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java:
##########
@@ -96,7 +97,8 @@ public void testGet() throws Exception {
keysStateKey(),
ByteArrayCoder.of(),
StringUtf8Coder.of(),
- true);
+ true,
+ false);
Review Comment:
Sideinput will not be affected by hasNoState now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2963078095
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -54,6 +55,8 @@ public class BagUserState<T> {
private List<T> newValues;
private boolean isCleared;
private boolean isClosed;
+ private final Supplier<Boolean> hasNoState;
Review Comment:
I believe all of these `*UserState` classes are only used for a single bundle, afterwhich they are closed and can't be used again. Hence we can simplify this and only pass in Boolean's. It's only the Factory that needs the Suppliers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6138993637
@ryan-mbuashundip Can you reopen this PR for me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2881209806
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java:
##########
@@ -246,7 +246,9 @@ private void addRunnerAndConsumersForPTransformRecursively(
BundleFinalizer bundleFinalizer,
Collection<BeamFnDataReadRunner<?>> channelRoots,
Map<ApiServiceDescriptor, BeamFnDataOutboundAggregator> outboundAggregatorMap,
- Set<String> runnerCapabilities)
+ Set<String> runnerCapabilities,
+ boolean hasNoState,
Review Comment:
Something to think about: do we need to worry about the case where `onlyBundleForKeys` is true and `hasNoState` is false? I think that should never happen. If that's true, then we should have a check somewhere in this whole stack (ideally as early as possible).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2963121174
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapUserState.java:
##########
@@ -93,14 +98,18 @@ public MultimapUserState(
this.beamFnStateClient = beamFnStateClient;
this.mapKeyCoder = mapKeyCoder;
this.valueCoder = valueCoder;
+ this.hasNoState = hasNoState;
+ this.onlyBundleForKeys = onlyBundleForKeys;
// Note: These StateRequest protos are constructed even if we never try to read the
// corresponding state type. Consider constructing them lazily, as needed.
Review Comment:
This seems even more true now. We don't need `this.userStateRequest` or `this.entryiesStateRequest` if `onlyBundleForKeys == true`. We could avoid creating the Builders and assigning them in this case. It looks like we still need `this.keysStateRequest` for now since it's used in a bunch of error logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "acrites (via GitHub)" <gi...@apache.org>.
acrites commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2978258770
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java:
##########
@@ -488,6 +492,16 @@ public BundleSplitListener getSplitListener() {
public BundleFinalizer getBundleFinalizer() {
return bundleFinalizer;
}
+
+ @Override
+ public Supplier<Boolean> getHasNoState() {
+ return () -> hasNoState;
Review Comment:
I still don't think this is quite right. `hasNoState` here is determined from the `ProcessBundleRequest` used when constructing the `BundleProcessor`. But we end up reusing these for other `ProcessBundleRequests`. You want to do something similar to how we handle `cacheTokens` in `setupForProcessBundleRequest`.
It would be good to write some kind of test that might catch this error. Something where we call `processBundle` for two different `InstructionRequests` that have the same `processBundleDescriptorId` (so we reuse the same `BundleProcessor` from the cache, but different values of these two new fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3950164212
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3947810328
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2908227339
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -677,6 +683,15 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient,
+ StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = hasNoState ? null : stateRequestForFirstChunk.getGet().getContinuationToken();
Review Comment:
Thanks for explanation. I see `hasNoState` semantically means `omitStates` here, it doesn't necessarily mean there is no state if we try to request for states at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #37691:
URL: https://github.com/apache/beam/pull/37691#discussion_r2854682144
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/OrderedListUserState.java:
##########
@@ -303,6 +309,11 @@ public void asyncClose() throws Exception {
pendingRemoves.clear();
}
+ if (onlyBundleForKeys) {
Review Comment:
Marking. another place where onlyBundleForKeys taking effect.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java:
##########
@@ -246,7 +246,9 @@ private void addRunnerAndConsumersForPTransformRecursively(
BundleFinalizer bundleFinalizer,
Collection<BeamFnDataReadRunner<?>> channelRoots,
Map<ApiServiceDescriptor, BeamFnDataOutboundAggregator> outboundAggregatorMap,
- Set<String> runnerCapabilities)
+ Set<String> runnerCapabilities,
Review Comment:
The number of parameters keep growing...
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java:
##########
@@ -127,7 +133,7 @@ public void asyncClose() throws Exception {
beamFnStateClient.handle(
request.toBuilder().setClear(StateClearRequest.getDefaultInstance()));
}
- if (!newValues.isEmpty()) {
+ if (!onlyBundleForKeys && !newValues.isEmpty()) {
Review Comment:
Marking. This is where the plumbed parameter taking effect.
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java:
##########
@@ -677,6 +683,15 @@ static class LazyBlockingStateFetchingIterator implements PrefetchableIterator<B
this.continuationToken = stateRequestForFirstChunk.getGet().getContinuationToken();
}
+ LazyBlockingStateFetchingIterator(
+ BeamFnStateClient beamFnStateClient,
+ StateRequest stateRequestForFirstChunk,
+ boolean hasNoState) {
+ this.beamFnStateClient = beamFnStateClient;
+ this.stateRequestForFirstChunk = stateRequestForFirstChunk;
+ this.continuationToken = hasNoState ? null : stateRequestForFirstChunk.getGet().getContinuationToken();
Review Comment:
This mostly LGTM me. The need of plumbing these parameters all the way down to where it takes effect tells how difficult this kind of changes in an existing system became.
My only concern is that we now unconditionally trust requests' hasNoState/onlyBundleForKey parameter to skip overheads. Is it possible to add some lightweight checks somewhere, for example, when we see there is actually states but hasNoState is set, we should throw IllegalArgumentException to prevent data loss / corruption when the assumption does not stand (in the case of runner bug)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3947535394
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-3947696072
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-4855411275
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6138642743
reopen the pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Omit state commits in stateful Dofn [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #37691:
URL: https://github.com/apache/beam/pull/37691#issuecomment-6161832709
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/37691?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `75.00000%` with `22 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.04%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`06a1365`](https://app.codecov.io/gh/apache/beam/commit/06a1365aacc84f242d0269b0dac65d60489f3d89?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 160 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/37691?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/beam/fn/harness/state/FnApiStateAccessor.java](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fstate%2FFnApiStateAccessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9zdGF0ZS9GbkFwaVN0YXRlQWNjZXNzb3IuamF2YQ==) | 45.45% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/beam/fn/harness/state/MultimapUserState.java](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fstate%2FMultimapUserState.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9zdGF0ZS9NdWx0aW1hcFVzZXJTdGF0ZS5qYXZh) | 81.81% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/beam/fn/harness/PTransformRunnerFactory.java](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2FPTransformRunnerFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9QVHJhbnNmb3JtUnVubmVyRmFjdG9yeS5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/beam/fn/harness/state/BagUserState.java](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fstate%2FBagUserState.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9zdGF0ZS9CYWdVc2VyU3RhdGUuamF2YQ==) | 88.23% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/beam/fn/harness/state/OrderedListUserState.java](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&filepath=sdks%2Fjava%2Fharness%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Ffn%2Fharness%2Fstate%2FOrderedListUserState.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9qYXZhL2hhcm5lc3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2JlYW0vZm4vaGFybmVzcy9zdGF0ZS9PcmRlcmVkTGlzdFVzZXJTdGF0ZS5qYXZh) | 80.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/37691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #37691 +/- ##
============================================
+ Coverage 56.16% 65.04% +8.87%
- Complexity 2288 13983 +11695
============================================
Files 1124 1730 +606
Lines 178336 108529 -69807
Branches 1489 11312 +9823
============================================
- Hits 100170 70596 -29574
+ Misses 75645 33723 -41922
- Partials 2521 4210 +1689
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/37691/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/37691/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java](https://app.codecov.io/gh/apache/beam/pull/37691/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.04% <75.00%> (-10.74%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/beam/pull/37691/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/37691?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org