You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bowenli86 <gi...@git.apache.org> on 2018/02/15 20:42:53 UTC

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

GitHub user bowenli86 opened a pull request:

    https://github.com/apache/flink/pull/5500

    [FLINK-8667] expose key in KeyedBroadcastProcessFunction#onTimer()

    ## What is the purpose of the change
    
    Expose key in `KeyedBroadcastProcessFunction#onTimer(OnTimerContext<Key>)`.
    
    Hopefully this PR can make it into 1.5.0, in which release`KeyedBroadcastProcessFunction` will be out. Otherwise, we need to think about compatibility and it will be much harder to expose key in `KeyedBroadcastProcessFunction#onTimer()`.
    
    ## Brief change log
    
    Expose key in `KeyedBroadcastProcessFunction#onTimer(OnTimerContext<Key>)`
    
    ## Verifying this change
    
    This change is already covered by existing tests, such as *BroadcastStateITCase*.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (yes)
      - If yes, how is the feature documented? (JavaDocs)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bowenli86/flink FLINK-8667

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5500
    
----
commit 2dcdedf072576e7e23fd220abf48c32c8355e5cc
Author: Bowen Li <bo...@...>
Date:   2018-02-15T20:37:44Z

    [FLINK-8667] expose key in KeyedBroadcastProcessFunction#onTimer()

----


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    The reason is what @tillrohrmann described. That also the other contexts do not have access to the key either. This is **partially** a design reason (for security), as we do not create defensive copies for performance reasons and if the user code modifies the key, then we may have issues, and **partially** something that we did not consider so important to expose in the beginning.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171252617
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
    @@ -118,7 +118,7 @@
     	 * @throws Exception This method may throw exceptions. Throwing an exception will cause the operation
     	 *                   to fail and may trigger recovery.
     	 */
    -	public void onTimer(final long timestamp, final OnTimerContext ctx, final Collector<OUT> out) throws Exception {
    +	public void onTimer(final long timestamp, final OnTimerContext<KS> ctx, final Collector<OUT> out) throws Exception {
    --- End diff --
    
    Remove the `<KS>`.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253807
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -847,12 +834,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
     				new KeyedBroadcastProcessFunction<String, Long, String, String>() {
     					@Override
     					public void processBroadcastElement(String value, KeyedContext ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169057332
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -169,7 +170,10 @@ public void open(Configuration parameters) throws Exception {
     
     		@Override
     		public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
    -			ctx.timerService().registerEventTimeTimer(timerTimestamp);
    +			long currentTime = al.getAndIncrement();
    --- End diff --
    
    @aljoscha I don't understand this. Shouldn't a user have an access to the `currentKey` in this method as well? There should be `ctx.getCurrentKey()` method. Otherwise we are forcing users to duplicate key selector code between here and the application definition.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171252476
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
    @@ -163,12 +163,16 @@ public void onTimer(final long timestamp, final OnTimerContext ctx, final Collec
     	/**
     	 * Information available in an invocation of {@link #onTimer(long, OnTimerContext, Collector)}.
     	 */
    -	public abstract class OnTimerContext extends KeyedReadOnlyContext {
    +	public abstract class OnTimerContext<KS> extends KeyedReadOnlyContext {
     
     		/**
    -		 * The {@link TimeDomain} of the firing timer, i.e. if it is
    -		 * event or processing time timer.
    +		 * The {@link TimeDomain} of the firing timer, i.e. if it is event or processing time timer.
     		 */
    --- End diff --
    
    Remove un-necessary reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169055575
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
     		private static final long serialVersionUID = 7616910653561100842L;
     
     		private final Map<Long, String> expectedState;
    -
    -		private final long timerTimestamp;
    +		private final AtomicLong al = new AtomicLong(1000L);
    +		// <key, time>
    +		private final Map<Long, Long> expectedTimeAndKey = new HashMap<>();
    +		// <time, key>
    +		private final Map<Long, Long> expectedKeyAndTime = new HashMap<>();
     
     		private transient MapStateDescriptor<Long, String> descriptor;
     
    -		TestBroadcastProcessFunction(
    -				final long timerTS,
    -				final Map<Long, String> expectedBroadcastState
    -		) {
    +		TestBroadcastProcessFunction(final Map<Long, String> expectedBroadcastState) {
    --- End diff --
    
    Please keep the `timerTS` (renaming it to `initialTimerTimestamp`) parameter as an initial value for the `nextTimerTimestamp`/`al` field that you have added. Otherwise it's confusing where does this magic value `1000` come from and what are consequences of changing it.  Especially that you have kept the original comment `// the timestamp should be high enough to trigger the timer after all the elements arrive.` in the `testConnectWithBroadcastTranslation()`.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169054889
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
     		private static final long serialVersionUID = 7616910653561100842L;
     
     		private final Map<Long, String> expectedState;
    -
    -		private final long timerTimestamp;
    +		private final AtomicLong al = new AtomicLong(1000L);
    --- End diff --
    
    Please rename `al` to `nextTimerTimestamp`


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    a irrelavent flaky unit test caused the error in travis build, and I've filed [FLINK-8709](https://issues.apache.org/jira/browse/FLINK-8709) for it


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169058054
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
     		private static final long serialVersionUID = 7616910653561100842L;
     
     		private final Map<Long, String> expectedState;
    -
    -		private final long timerTimestamp;
    +		private final AtomicLong al = new AtomicLong(1000L);
    +		// <key, time>
    +		private final Map<Long, Long> expectedTimeAndKey = new HashMap<>();
    --- End diff --
    
    I think the name `timerToExpectedKey` would better express the intention and the comment `// <key, time>` is redundant.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253604
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -740,24 +734,17 @@ public void testProcessTranslation() {
     			private static final long serialVersionUID = 1L;
     
     			@Override
    -			public void processElement(
    -					Long value,
    -					Context ctx,
    -					Collector<Integer> out) throws Exception {
    -
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169055105
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -145,17 +147,16 @@ public Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp
     		private static final long serialVersionUID = 7616910653561100842L;
     
     		private final Map<Long, String> expectedState;
    -
    -		private final long timerTimestamp;
    +		private final AtomicLong al = new AtomicLong(1000L);
    --- End diff --
    
    Is this `AtomicLong` necessary? As it is now, it suggests some concurrent access and thus is confusing.


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    @pnowojski @aljoscha  I updated the code. Hopefully we can make this into 1.5.0! Thanks!


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    cc @aljoscha  @pnowojski 


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253672
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -801,12 +788,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
     				new BroadcastProcessFunction<Long, String, String>() {
     					@Override
     					public void processBroadcastElement(String value, Context ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    cc @pnowojski @aljoscha 


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169057826
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BroadcastStateITCase.java ---
    @@ -169,7 +170,10 @@ public void open(Configuration parameters) throws Exception {
     
     		@Override
     		public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
    -			ctx.timerService().registerEventTimeTimer(timerTimestamp);
    +			long currentTime = al.getAndIncrement();
    +			ctx.timerService().registerEventTimeTimer(currentTime);
    +			expectedTimeAndKey.put(currentTime, value);
    +			expectedKeyAndTime.put(value, currentTime);
    --- End diff --
    
    What's the value of this `expectedKeyAndTime` mapping? because incrementing of the `al` counter, keys in the first map (`expectedTimeAndKey`) will always be unique and I think we can safely remove this second map.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253275
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
    @@ -381,26 +392,28 @@ public void processElement(String value, KeyedReadOnlyContext ctx, Collector<Str
     			Iterator<Map.Entry<String, Integer>> iter = broadcastStateIt.iterator();
     
     			for (int i = 0; i < expectedBroadcastState.size(); i++) {
    -				Assert.assertTrue(iter.hasNext());
    +				assertTrue(iter.hasNext());
     
     				Map.Entry<String, Integer> entry = iter.next();
    -				Assert.assertTrue(expectedBroadcastState.containsKey(entry.getKey()));
    -				Assert.assertEquals(expectedBroadcastState.get(entry.getKey()), entry.getValue());
    +				assertTrue(expectedBroadcastState.containsKey(entry.getKey()));
    +				assertEquals(expectedBroadcastState.get(entry.getKey()), entry.getValue());
     			}
     
    -			Assert.assertFalse(iter.hasNext());
    +			assertFalse(iter.hasNext());
     
     			ctx.timerService().registerEventTimeTimer(timerTs);
     		}
     
     		@Override
    -		public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
    +		public void onTimer(long timestamp, OnTimerContext<String> ctx, Collector<String> out) throws Exception {
    --- End diff --
    
    Remove the `<String>`.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253628
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -740,24 +734,17 @@ public void testProcessTranslation() {
     			private static final long serialVersionUID = 1L;
     
     			@Override
    -			public void processElement(
    -					Long value,
    -					Context ctx,
    -					Collector<Integer> out) throws Exception {
    -
    +			public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
    +				// Do nothing
     			}
     
     			@Override
    -			public void onTimer(
    -					long timestamp,
    -					OnTimerContext ctx,
    -					Collector<Integer> out) throws Exception {
    -
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    LGTM! Thanks for the work @bowenli86 !


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    @tillrohrmann  @kl0u Thanks for reviewing, guys 
    
    As @pnowojski mentioned, we three decided to expose timer keys in `ProcessFunction` in [FLINK-8560](https://github.com/apache/flink/pull/5481).  Exposing timer keys in `KeyedBroadcastProcessFunction` extends that design. I think we should get this PR into 1.5.0 so we don't need to do the [complicated refactoring for FLINK-8560](https://github.com/apache/flink/pull/5481) to support backward compatibility 


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253550
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -701,19 +701,13 @@ public void testKeyedProcessTranslation() {
     			private static final long serialVersionUID = 1L;
     
     			@Override
    -			public void processElement(
    -					Long value,
    -					Context ctx,
    -					Collector<Integer> out) throws Exception {
    -
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253575
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -701,19 +701,13 @@ public void testKeyedProcessTranslation() {
     			private static final long serialVersionUID = 1L;
     
     			@Override
    -			public void processElement(
    -					Long value,
    -					Context ctx,
    -					Collector<Integer> out) throws Exception {
    -
    +			public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
    +				// Do nothing
     			}
     
     			@Override
    -			public void onTimer(
    -					long timestamp,
    -					OnTimerContext ctx,
    -					Collector<Integer> out) throws Exception {
    -
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171252831
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
    @@ -288,7 +288,7 @@ public long currentWatermark() {
     		}
     	}
     
    -	private class OnTimerContextImpl extends KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT>.OnTimerContext {
    +	private class OnTimerContextImpl<KS> extends KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT>.OnTimerContext<KS> {
     
    --- End diff --
    
    Remove the `<KS>` in both places.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169051920
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
    @@ -204,6 +208,8 @@ public void testFunctionWithTimer() throws Exception {
     
     		private final long timerTS;
     
    +		static final String EXPECTED_KEY = "6";
    --- End diff --
    
    Can not this be a non static field initialised in the constructor just as `timerTS` is? 


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253650
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -740,24 +734,17 @@ public void testProcessTranslation() {
     			private static final long serialVersionUID = 1L;
     
     			@Override
    -			public void processElement(
    -					Long value,
    -					Context ctx,
    -					Collector<Integer> out) throws Exception {
    -
    +			public void processElement(Long value, Context ctx, Collector<Integer> out) throws Exception {
    +				// Do nothing
     			}
     
     			@Override
    -			public void onTimer(
    -					long timestamp,
    -					OnTimerContext ctx,
    -					Collector<Integer> out) throws Exception {
    -
    +			public void onTimer(long timestamp, OnTimerContext ctx, Collector<Integer> out) throws Exception {
    +				// Do nothing
     			}
     		};
     
    -		DataStream<Integer> processed = src
    -				.process(processFunction);
    +		DataStream<Integer> processed = src.process(processFunction);
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253126
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
    @@ -220,7 +227,8 @@ public void processElement(String value, KeyedReadOnlyContext ctx, Collector<Str
     		}
     
     		@Override
    -		public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
    +		public void onTimer(long timestamp, OnTimerContext<String> ctx, Collector<String> out) throws Exception {
    --- End diff --
    
    Remove the `<String>`.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5500


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253705
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -801,12 +788,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
     				new BroadcastProcessFunction<Long, String, String>() {
     					@Override
     					public void processBroadcastElement(String value, Context ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
     					}
     
     					@Override
     					public void processElement(Long value, ReadOnlyContext ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r168855562
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
    @@ -324,6 +324,11 @@ public TimeDomain timeDomain() {
     			return timeDomain;
     		}
     
    +		@Override
    +		public KS getCurrentKey() {
    --- End diff --
    
    added


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    If also @aljoscha is ok, I can review this PR and merge it.



---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    Hi @aljoscha , can you take a look? 


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r169051991
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperatorTest.java ---
    @@ -358,6 +365,8 @@ public void testFunctionWithBroadcastState() throws Exception {
     		private final Map<String, Integer> expectedBroadcastState;
     		private final long timerTs;
     
    +		static final String EXPECTED_KEY = "trigger";
    --- End diff --
    
    same here.


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    If this security concern is an issue, maybe in that case `getCurrentKey()` should take `object-reuse` into account, and if `object-reuse` is set to false, make a defensive copy of the key on access?
    
    However @aljoscha was not objecting the general idea in the other PR: https://github.com/apache/flink/pull/5481


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r168697719
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/CoBroadcastWithKeyedOperator.java ---
    @@ -324,6 +324,11 @@ public TimeDomain timeDomain() {
     			return timeDomain;
     		}
     
    +		@Override
    +		public KS getCurrentKey() {
    --- End diff --
    
    It needs at least some most basic test coverage


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171253833
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/DataStreamTest.java ---
    @@ -847,12 +834,12 @@ public long extractTimestamp(String element, long previousElementTimestamp) {
     				new KeyedBroadcastProcessFunction<String, Long, String, String>() {
     					@Override
     					public void processBroadcastElement(String value, KeyedContext ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
     					}
     
     					@Override
     					public void processElement(Long value, KeyedReadOnlyContext ctx, Collector<String> out) throws Exception {
    -						// do nothing
    +						// Do nothing
    --- End diff --
    
    Remove reformatting.


---

[GitHub] flink pull request #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFu...

Posted by kl0u <gi...@git.apache.org>.
Github user kl0u commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5500#discussion_r171252385
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/KeyedBroadcastProcessFunction.java ---
    @@ -163,12 +163,16 @@ public void onTimer(final long timestamp, final OnTimerContext ctx, final Collec
     	/**
     	 * Information available in an invocation of {@link #onTimer(long, OnTimerContext, Collector)}.
     	 */
    -	public abstract class OnTimerContext extends KeyedReadOnlyContext {
    +	public abstract class OnTimerContext<KS> extends KeyedReadOnlyContext {
    --- End diff --
    
    You can remove the `<KS>` as done in https://github.com/apache/flink/pull/5481 


---

[GitHub] flink issue #5500: [FLINK-8667] expose key in KeyedBroadcastProcessFunction#...

Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:

    https://github.com/apache/flink/pull/5500
  
    @kl0u @aljoscha I've updated this PR, and its build is green


---