You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by Renkai <gi...@git.apache.org> on 2016/08/11 12:56:44 UTC
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
GitHub user Renkai opened a pull request:
https://github.com/apache/flink/pull/2355
[FLINK-4282]Add Offset Parameter to WindowAssigners
Although there is already a merge request for this issue,I think this implementation is more sensible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Renkai/flink FLINK-4282
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2355.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2355
----
commit d31cc8125b30212b6ac21996a48d703eb11354e9
Author: renkai <ga...@gmail.com>
Date: 2016-08-11T10:48:50Z
[FLINK-4282]Add Offset Parameter to WindowAssigners
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2355: [FLINK-4282]Add Offset Parameter to WindowAssigners
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2355
I merged it. Thanks for your work! \U0001f603
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75306425
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowingTestHarnessTest.java ---
@@ -84,6 +86,84 @@ public void testEventTimeTumblingWindows() throws Exception {
}
@Test
+ public void testEventTimeTumblingWindowsWithOffset() throws Exception {
--- End diff --
These tests should go into `WindowOperatorTest`. The existing tests here are just meant to exercise the `WindowTestHarness`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by soniclavier <gi...@git.apache.org>.
Github user soniclavier commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r86656836
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/windows/TimeWindow.java ---
@@ -236,6 +236,6 @@ public int compare(TimeWindow o1, TimeWindow o2) {
* @return window start
*/
public static long getWindowStartWithOffset(long timestamp, long offset, long windowSize) {
--- End diff --
Is the windowSize actually slide size? I see that it is called by passing the slide instead of size `TimeWindow.getWindowStartWithOffset(timestamp, offset, slide);`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2355: [FLINK-4282]Add Offset Parameter to WindowAssigners
Posted by Renkai <gi...@git.apache.org>.
Github user Renkai commented on the issue:
https://github.com/apache/flink/pull/2355
I have rearranged the code. It is really hard for me to add space after every comma,do you have any code style preset for IDEs so that I can reformat my new code automatically without change those of others?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r86749516
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/windows/TimeWindow.java ---
@@ -236,6 +236,6 @@ public int compare(TimeWindow o1, TimeWindow o2) {
* @return window start
*/
public static long getWindowStartWithOffset(long timestamp, long offset, long windowSize) {
--- End diff --
In case of sliding windows we have to pass the slide here, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75305677
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/SlidingEventTimeWindows.java ---
@@ -52,16 +52,19 @@
private final long slide;
- protected SlidingEventTimeWindows(long size, long slide) {
+ private final long offset;
+
+ protected SlidingEventTimeWindows(long size, long slide,long offset) {
this.size = size;
this.slide = slide;
+ this.offset = offset;
}
@Override
public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) {
if (timestamp > Long.MIN_VALUE) {
List<TimeWindow> windows = new ArrayList<>((int) (size / slide));
- long lastStart = timestamp - timestamp % slide;
+ long lastStart = TimeWindow.getWindowStartWithOffset(timestamp,offset,slide);
--- End diff --
Missing spaces after comma.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75305993
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java ---
@@ -553,7 +553,7 @@ public void testSessionWindowsWithCountTrigger() throws Exception {
new OneInputStreamOperatorTestHarness<>(operator);
testHarness.configureForKeyedStream(new TupleKeySelector(), BasicTypeInfo.STRING_TYPE_INFO);
-
+
--- End diff --
Unrelated whitespace change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2355
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75305729
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/SlidingEventTimeWindows.java ---
@@ -102,7 +105,32 @@ public String toString() {
* @return The time policy.
*/
public static SlidingEventTimeWindows of(Time size, Time slide) {
- return new SlidingEventTimeWindows(size.toMilliseconds(), slide.toMilliseconds());
+ return new SlidingEventTimeWindows(size.toMilliseconds(), slide.toMilliseconds(),0);
+ }
+
+ /**
+ * Creates a new {@code SlidingEventTimeWindows} {@link WindowAssigner} that assigns
+ * elements to time windows based on the element timestamp and offset.
+ *<p>
+ * For example, if you want window a stream by hour,but window begins at the 15th minutes
+ * of each hour, you can use {@code of(Time.hours(1),Time.minutes(15))},then you will get
--- End diff --
Missing space after comma.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2355: [FLINK-4282]Add Offset Parameter to WindowAssigners
Posted by Renkai <gi...@git.apache.org>.
Github user Renkai commented on the issue:
https://github.com/apache/flink/pull/2355
@aljoscha It's fine.I'll try to write well formatted code manually from now on, and I hope the code in the last commit is well formatted enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75306168
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java ---
@@ -2550,4 +2550,36 @@ public String toString() {
return "EventTimeTrigger()";
}
}
+
+ @Test
--- End diff --
Very good that you added a test for this! \U0001f44d
I'd like to have these in a separate test for `TimeWindow`, however, since the `WindowOperatorTest` file is already very big.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2355: [FLINK-4282]Add Offset Parameter to WindowAssigners
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2355
@Renkai Unfortunately I'm not aware of such a tool. The IDE I use (IntelliJ) tends to always do changes in a lot of other places when I use the reformat tool so I try to stay away from that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2355: [FLINK-4282]Add Offset Parameter to WindowAssigner...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2355#discussion_r75305650
--- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/SlidingEventTimeWindows.java ---
@@ -52,16 +52,19 @@
private final long slide;
- protected SlidingEventTimeWindows(long size, long slide) {
+ private final long offset;
+
+ protected SlidingEventTimeWindows(long size, long slide,long offset) {
--- End diff --
Missing space before `long offset`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2355: [FLINK-4282]Add Offset Parameter to WindowAssigners
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2355
The code looks very good! I had some comments about whitespace/formatting and the placement of tests. (It's quite hard to know where to put these without a lot of experience with the code base.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---