You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bowenli86 <gi...@git.apache.org> on 2018/02/15 20:42:53 UTC
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
GitHub user bowenli86 opened a pull request:
https://github.com/apache/flink/pull/5500
[FLINK-8667] expose key in KeyedBroadcastProcessFunction#onTimer()
## What is the purpose of the change
Expose key in `KeyedBroadcastProcessFunction#onTimer(OnTimerContext<Key>)`.
Hopefully this PR can make it into 1.5.0, in which release`KeyedBroadcastProcessFunction` will be out. Otherwise, we need to think about compatibility and it will be much harder to expose key in `KeyedBroadcastProcessFunction#onTimer()`.
## Brief change log
Expose key in `KeyedBroadcastProcessFunction#onTimer(OnTimerContext<Key>)`
## Verifying this change
This change is already covered by existing tests, such as *BroadcastStateITCase*.
## Does this pull request potentially affect one of the following parts:
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-8667
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5500.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5500
----
commit 2dcdedf072576e7e23fd220abf48c32c8355e5cc
Author: Bowen Li <bo...@...>
Date: 2018-02-15T20:37:44Z
[FLINK-8667] expose key in KeyedBroadcastProcessFunction#onTimer()
----
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/5500
The reason is what @tillrohrmann described. That also the other contexts do not have access to the key either. This is **partially** a design reason (for security), as we do not create defensive copies for performance reasons and if the user code modifies the key, then we may have issues, and **partially** something that we did not consider so important to expose in the beginning.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171252617
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
@@ -118,7 +118,7 @@
* @throws Exception This method may throw exceptions. Throwing an exception will cause the operation
* to fail and may trigger recovery.
*/
- public void onTimer(final long timestamp, final OnTimerContext ctx, final Collector<OUT> out) throws Exception {
+ public void onTimer(final long timestamp, final OnTimerContext<KS> ctx, final Collector<OUT> out) throws Exception {
--- End diff --
Remove the `<KS>`.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253807
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -847,12 +834,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
new KeyedBroadcastProcessFunction<String, Long, String, String>() {
@Override
public void processBroadcastElement(String value, KeyedContext ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169057332
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -169,7 +170,10 @@ public void open(Configuration parameters) throws Exception {
@Override
public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
- ctx.timerService().registerEventTimeTimer(timerTimestamp);
+ long currentTime = al.getAndIncrement();
--- End diff --
@aljoscha I don't understand this. Shouldn't a user have an access to the `currentKey` in this method as well? There should be `ctx.getCurrentKey()` method. Otherwise we are forcing users to duplicate key selector code between here and the application definition.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171252476
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
@@ -163,12 +163,16 @@ public void onTimer(final long timestamp, final OnTimerContext ctx, final Collec
/**
* Information available in an invocation of {@link #onTimer(long, OnTimerContext, Collector)}.
*/
- public abstract class OnTimerContext extends KeyedReadOnlyContext {
+ public abstract class OnTimerContext<KS> extends KeyedReadOnlyContext {
/**
- * The {@link TimeDomain} of the firing timer, i.e. if it is
- * event or processing time timer.
+ * The {@link TimeDomain} of the firing timer, i.e. if it is event or processing time timer.
*/
--- End diff --
Remove un-necessary reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169055575
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
private static final long serialVersionUID = 7616910653561100842L;
private final Map<Long, String> expectedState;
-
- private final long timerTimestamp;
+ private final AtomicLong al = new AtomicLong(1000L);
+ // <key, time>
+ private final Map<Long, Long> expectedTimeAndKey = new HashMap<>();
+ // <time, key>
+ private final Map<Long, Long> expectedKeyAndTime = new HashMap<>();
private transient MapStateDescriptor<Long, String> descriptor;
- TestBroadcastProcessFunction(
- final long timerTS,
- final Map<Long, String> expectedBroadcastState
- ) {
+ TestBroadcastProcessFunction(final Map<Long, String> expectedBroadcastState) {
--- End diff --
Please keep the `timerTS` (renaming it to `initialTimerTimestamp`) parameter as an initial value for the `nextTimerTimestamp`/`al` field that you have added. Otherwise it's confusing where does this magic value `1000` come from and what are consequences of changing it. Especially that you have kept the original comment `// the timestamp should be high enough to trigger the timer after all the elements arrive.` in the `testConnectWithBroadcastTranslation()`.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169054889
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
private static final long serialVersionUID = 7616910653561100842L;
private final Map<Long, String> expectedState;
-
- private final long timerTimestamp;
+ private final AtomicLong al = new AtomicLong(1000L);
--- End diff --
Please rename `al` to `nextTimerTimestamp`
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
a irrelavent flaky unit test caused the error in travis build, and I've filed [FLINK-8709](https://issues.apache.org/jira/browse/FLINK-8709) for it
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169058054
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
private static final long serialVersionUID = 7616910653561100842L;
private final Map<Long, String> expectedState;
-
- private final long timerTimestamp;
+ private final AtomicLong al = new AtomicLong(1000L);
+ // <key, time>
+ private final Map<Long, Long> expectedTimeAndKey = new HashMap<>();
--- End diff --
I think the name `timerToExpectedKey` would better express the intention and the comment `// <key, time>` is redundant.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253604
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -740,24 +734,17 @@ public void testProcessTranslation() {
private static final long serialVersionUID = 1L;
@Override
- public void processElement(
- Long value,
- Context ctx,
- Collector<Integer> out) throws Exception {
-
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169055105
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
private static final long serialVersionUID = 7616910653561100842L;
private final Map<Long, String> expectedState;
-
- private final long timerTimestamp;
+ private final AtomicLong al = new AtomicLong(1000L);
--- End diff --
Is this `AtomicLong` necessary? As it is now, it suggests some concurrent access and thus is confusing.
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
@pnowojski @aljoscha I updated the code. Hopefully we can make this into 1.5.0! Thanks!
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
cc @aljoscha @pnowojski
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253672
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -801,12 +788,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
new BroadcastProcessFunction<Long, String, String>() {
@Override
public void processBroadcastElement(String value, Context ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
--- End diff --
Remove reformatting.
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
cc @pnowojski @aljoscha
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169057826
--- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
@@ -169,7 +170,10 @@ public void open(Configuration parameters) throws Exception {
@Override
public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
- ctx.timerService().registerEventTimeTimer(timerTimestamp);
+ long currentTime = al.getAndIncrement();
+ ctx.timerService().registerEventTimeTimer(currentTime);
+ expectedTimeAndKey.put(currentTime, value);
+ expectedKeyAndTime.put(value, currentTime);
--- End diff --
What's the value of this `expectedKeyAndTime` mapping? because incrementing of the `al` counter, keys in the first map (`expectedTimeAndKey`) will always be unique and I think we can safely remove this second map.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253275
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
@@ -381,26 +392,28 @@ public void processElement(String value, KeyedReadOnlyContext ctx, Collector<Str
Iterator<Map.Entry<String, Integer>> iter = broadcastStateIt.iterator();
for (int i = 0; i < expectedBroadcastState.size(); i++) {
- Assert.assertTrue(iter.hasNext());
+ assertTrue(iter.hasNext());
Map.Entry<String, Integer> entry = iter.next();
- Assert.assertTrue(expectedBroadcastState.containsKey(entry.getKey()));
- Assert.assertEquals(expectedBroadcastState.get(entry.getKey()), entry.getValue());
+ assertTrue(expectedBroadcastState.containsKey(entry.getKey()));
+ assertEquals(expectedBroadcastState.get(entry.getKey()), entry.getValue());
}
- Assert.assertFalse(iter.hasNext());
+ assertFalse(iter.hasNext());
ctx.timerService().registerEventTimeTimer(timerTs);
}
@Override
- public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
+ public void onTimer(long timestamp, OnTimerContext<String> ctx, Collector<String> out) throws Exception {
--- End diff --
Remove the `<String>`.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253628
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -740,24 +734,17 @@ public void testProcessTranslation() {
private static final long serialVersionUID = 1L;
@Override
- public void processElement(
- Long value,
- Context ctx,
- Collector<Integer> out) throws Exception {
-
+ public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
+ // Do nothing
}
@Override
- public void onTimer(
- long timestamp,
- OnTimerContext ctx,
- Collector<Integer> out) throws Exception {
-
--- End diff --
Remove reformatting.
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/5500
LGTM! Thanks for the work @bowenli86 !
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
@tillrohrmann @kl0u Thanks for reviewing, guys
As @pnowojski mentioned, we three decided to expose timer keys in `ProcessFunction` in [FLINK-8560](https://github.com/apache/flink/pull/5481). Exposing timer keys in `KeyedBroadcastProcessFunction` extends that design. I think we should get this PR into 1.5.0 so we don't need to do the [complicated refactoring for FLINK-8560](https://github.com/apache/flink/pull/5481) to support backward compatibility
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253550
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -701,19 +701,13 @@ public void testKeyedProcessTranslation() {
private static final long serialVersionUID = 1L;
@Override
- public void processElement(
- Long value,
- Context ctx,
- Collector<Integer> out) throws Exception {
-
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253575
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -701,19 +701,13 @@ public void testKeyedProcessTranslation() {
private static final long serialVersionUID = 1L;
@Override
- public void processElement(
- Long value,
- Context ctx,
- Collector<Integer> out) throws Exception {
-
+ public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
+ // Do nothing
}
@Override
- public void onTimer(
- long timestamp,
- OnTimerContext ctx,
- Collector<Integer> out) throws Exception {
-
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171252831
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
@@ -288,7 +288,7 @@ public long currentWatermark() {
}
}
- private class OnTimerContextImpl extends KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT>.OnTimerContext {
+ private class OnTimerContextImpl<KS> extends KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT>.OnTimerContext<KS> {
--- End diff --
Remove the `<KS>` in both places.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169051920
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
@@ -204,6 +208,8 @@ public void testFunctionWithTimer() throws Exception {
private final long timerTS;
+ static final String EXPECTED_KEY = "6";
--- End diff --
Can not this be a non static field initialised in the constructor just as `timerTS` is?
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253650
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -740,24 +734,17 @@ public void testProcessTranslation() {
private static final long serialVersionUID = 1L;
@Override
- public void processElement(
- Long value,
- Context ctx,
- Collector<Integer> out) throws Exception {
-
+ public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
+ // Do nothing
}
@Override
- public void onTimer(
- long timestamp,
- OnTimerContext ctx,
- Collector<Integer> out) throws Exception {
-
+ public void onTimer(long timestamp, OnTimerContext ctx, Collector<Integer> out) throws Exception {
+ // Do nothing
}
};
- DataStream<Integer> processed = src
- .process(processFunction);
+ DataStream<Integer> processed = src.process(processFunction);
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253126
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
@@ -220,7 +227,8 @@ public void processElement(String value, KeyedReadOnlyContext ctx, Collector<Str
}
@Override
- public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
+ public void onTimer(long timestamp, OnTimerContext<String> ctx, Collector<String> out) throws Exception {
--- End diff --
Remove the `<String>`.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5500
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253705
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -801,12 +788,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
new BroadcastProcessFunction<Long, String, String>() {
@Override
public void processBroadcastElement(String value, Context ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
}
@Override
public void processElement(Long value, ReadOnlyContext ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r168855562
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
@@ -324,6 +324,11 @@ public TimeDomain timeDomain() {
return timeDomain;
}
+ @Override
+ public KS getCurrentKey() {
--- End diff --
added
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/5500
If also @aljoscha is ok, I can review this PR and merge it.
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
Hi @aljoscha , can you take a look?
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r169051991
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
@@ -358,6 +365,8 @@ public void testFunctionWithBroadcastState() throws Exception {
private final Map<String, Integer> expectedBroadcastState;
private final long timerTs;
+ static final String EXPECTED_KEY = "trigger";
--- End diff --
same here.
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:
https://github.com/apache/flink/pull/5500
If this security concern is an issue, maybe in that case `getCurrentKey()` should take `object-reuse` into account, and if `object-reuse` is set to false, make a defensive copy of the key on access?
However @aljoscha was not objecting the general idea in the other PR: https://github.com/apache/flink/pull/5481
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r168697719
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
@@ -324,6 +324,11 @@ public TimeDomain timeDomain() {
return timeDomain;
}
+ @Override
+ public KS getCurrentKey() {
--- End diff --
It needs at least some most basic test coverage
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171253833
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
@@ -847,12 +834,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
new KeyedBroadcastProcessFunction<String, Long, String, String>() {
@Override
public void processBroadcastElement(String value, KeyedContext ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
}
@Override
public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
- // do nothing
+ // Do nothing
--- End diff --
Remove reformatting.
---
[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...
Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:
https://github.com/apache/flink/pull/5500#discussion_r171252385
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
@@ -163,12 +163,16 @@ public void onTimer(final long timestamp, final OnTimerContext ctx, final Collec
/**
* Information available in an invocation of {@link #onTimer(long, OnTimerContext, Collector)}.
*/
- public abstract class OnTimerContext extends KeyedReadOnlyContext {
+ public abstract class OnTimerContext<KS> extends KeyedReadOnlyContext {
--- End diff --
You can remove the `<KS>` as done in https://github.com/apache/flink/pull/5481
---
[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/5500
@kl0u @aljoscha I've updated this PR, and its build is green
---