You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/10/12 23:31:54 UTC
[GitHub] [beam] lukecwik commented on a change in pull request #15704: [BEAM-12922] Test MapState putIfAbsent with no read
lukecwik commented on a change in pull request #15704:
URL: https://github.com/apache/beam/pull/15704#discussion_r727581454
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##########
@@ -2438,6 +2438,62 @@ public void processElement(
pipeline.run();
}
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesStatefulParDo.class,
+ UsesMapState.class,
+ UsesTestStream.class
+ })
+ public void testMapStateNoReadOnReadableState() {
+ final String stateId = "foo";
+ final String countStateId = "count";
+
+ DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>> fn =
+ new DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>>() {
+
+ @StateId(stateId)
+ private final StateSpec<MapState<String, Integer>> mapState =
+ StateSpecs.map(StringUtf8Coder.of(), VarIntCoder.of());
+
+ @StateId(countStateId)
+ private final StateSpec<CombiningState<Integer, int[], Integer>> countState =
+ StateSpecs.combiningFromInputInternal(VarIntCoder.of(), Sum.ofIntegers());
Review comment:
Is coder inference broken if we rely on `combining`?
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##########
@@ -2438,6 +2438,62 @@ public void processElement(
pipeline.run();
}
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesStatefulParDo.class,
+ UsesMapState.class,
+ UsesTestStream.class
+ })
+ public void testMapStateNoReadOnReadableState() {
+ final String stateId = "foo";
+ final String countStateId = "count";
+
+ DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>> fn =
+ new DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>>() {
+
+ @StateId(stateId)
+ private final StateSpec<MapState<String, Integer>> mapState =
+ StateSpecs.map(StringUtf8Coder.of(), VarIntCoder.of());
+
+ @StateId(countStateId)
+ private final StateSpec<CombiningState<Integer, int[], Integer>> countState =
+ StateSpecs.combiningFromInputInternal(VarIntCoder.of(), Sum.ofIntegers());
+
+ @ProcessElement
+ public void processElement(
+ ProcessContext c,
+ @Element KV<String, KV<String, Integer>> element,
+ @StateId(stateId) MapState<String, Integer> state,
+ @StateId(countStateId) CombiningState<Integer, int[], Integer> count,
+ OutputReceiver<KV<String, Integer>> r) {
+ KV<String, Integer> value = element.getValue();
+ state.putIfAbsent(value.getKey(), value.getValue());
Review comment:
Could we use putIfAbsent() for `a` and computeIfAbsent for `b` so we get coverage for both?
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##########
@@ -2438,6 +2438,62 @@ public void processElement(
pipeline.run();
}
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesStatefulParDo.class,
+ UsesMapState.class,
+ UsesTestStream.class
+ })
+ public void testMapStateNoReadOnReadableState() {
+ final String stateId = "foo";
+ final String countStateId = "count";
+
+ DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>> fn =
+ new DoFn<KV<String, KV<String, Integer>>, KV<String, Integer>>() {
+
+ @StateId(stateId)
+ private final StateSpec<MapState<String, Integer>> mapState =
+ StateSpecs.map(StringUtf8Coder.of(), VarIntCoder.of());
+
+ @StateId(countStateId)
+ private final StateSpec<CombiningState<Integer, int[], Integer>> countState =
+ StateSpecs.combiningFromInputInternal(VarIntCoder.of(), Sum.ofIntegers());
+
+ @ProcessElement
+ public void processElement(
+ ProcessContext c,
+ @Element KV<String, KV<String, Integer>> element,
+ @StateId(stateId) MapState<String, Integer> state,
+ @StateId(countStateId) CombiningState<Integer, int[], Integer> count,
+ OutputReceiver<KV<String, Integer>> r) {
+ KV<String, Integer> value = element.getValue();
+ state.putIfAbsent(value.getKey(), value.getValue());
+ count.add(1);
+ if (count.read() >= 4) {
+ Iterable<Map.Entry<String, Integer>> iterate = state.entries().read();
+ assertEquals(2, Iterables.size(iterate));
Review comment:
```suggestion
```
This is redundant when compared with the PAssert that runs outside of the pipeline.
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##########
@@ -2438,6 +2438,62 @@ public void processElement(
pipeline.run();
}
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesStatefulParDo.class,
+ UsesMapState.class,
+ UsesTestStream.class
+ })
+ public void testMapStateNoReadOnReadableState() {
Review comment:
```suggestion
public void testMapStateNoReadOnComputeIfAbsentAndPutIfAbsentInsertsElement() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org