You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/11 15:24:05 UTC
[GitHub] [flink] beyond1920 opened a new pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
beyond1920 opened a new pull request #14610:
URL: https://github.com/apache/flink/pull/14610
## What is the purpose of the change
MiniBatch infer does not work well if job contains unbounded aggregate and deduplicate because deduplicate does not take rowtime mode into consideration when implements `requireWatermark` method.
## Brief change log
Minor update on `StreamExecDeduplicate`
## Verifying this change
UT in `DeduplicateTest` and IT in `DeduplicateITCase`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
}, {
"hash" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12135",
"triggerID" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0482293fe8ca9fdd51defaa6db8e86b35dac1c5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12135)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r555490717
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
we should enable `minibatch` here ? and we can also add a test for firstRow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r555537296
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
hah, you're wright @beyond1920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9f9cf0caee79ac6382288b98ec98cad50a706cc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067)
* 384aa1d095bc02c376ec2a0af43501a19e2a81ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
beyond1920 commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r558281721
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/DeduplicateTest.scala
##########
@@ -141,6 +162,27 @@ class DeduplicateTest extends TableTestBase {
util.verifyExecPlan(sql)
}
+ @Test
+ def testMiniBatchInferLastRowOnRowtime(): Unit = {
+ util.tableEnv.getConfig.getConfiguration.setBoolean(TABLE_EXEC_MINIBATCH_ENABLED, true)
+ util.tableEnv.getConfig.getConfiguration.setLong(TABLE_EXEC_MINIBATCH_SIZE, 3L)
+ util.tableEnv.getConfig.getConfiguration.set(
+ TABLE_EXEC_MINIBATCH_ALLOW_LATENCY, Duration.ofSeconds(1))
+ val sql =
+ """
+ |SELECT COUNT(b) FROM (
+ | SELECT a, b, c
+ | FROM (
+ | SELECT *,
+ | ROW_NUMBER() OVER (PARTITION BY a ORDER BY rowtime DESC) as rank_num
+ | FROM MyTable)
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73247f65a10d8a1e4df8608044c7c9c811505520 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890)
* c9f9cf0caee79ac6382288b98ec98cad50a706cc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
beyond1920 commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r557475329
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
OK，done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong merged pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
wuchong merged pull request #14610:
URL: https://github.com/apache/flink/pull/14610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
beyond1920 commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r556214065
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
yes. DeduplicateTest is to test plan is same as expected when `Deduplicate` is followed by unbounded agg, DeduplicateITCase is to test result is same as expected in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
}, {
"hash" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12135",
"triggerID" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384aa1d095bc02c376ec2a0af43501a19e2a81ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123)
* 0482293fe8ca9fdd51defaa6db8e86b35dac1c5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12135)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
beyond1920 commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r555497127
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
@leonardBang Thanks for suggestion. DeduplicateITCase already tests four types of configuration for each case:
MiniBatchOn + Heap, MiniBatchOn + RocksDB, MiniBatchOff + Heap, MiniBatchOff + RocksDB, so I don't configure miniBatch in this case.
I could add a test for FirstRow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
}, {
"hash" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0482293fe8ca9fdd51defaa6db8e86b35dac1c5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384aa1d095bc02c376ec2a0af43501a19e2a81ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123)
* 0482293fe8ca9fdd51defaa6db8e86b35dac1c5f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9f9cf0caee79ac6382288b98ec98cad50a706cc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067)
* 384aa1d095bc02c376ec2a0af43501a19e2a81ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73247f65a10d8a1e4df8608044c7c9c811505520 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73247f65a10d8a1e4df8608044c7c9c811505520 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r558115913
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/DeduplicateTest.scala
##########
@@ -141,6 +162,27 @@ class DeduplicateTest extends TableTestBase {
util.verifyExecPlan(sql)
}
+ @Test
+ def testMiniBatchInferLastRowOnRowtime(): Unit = {
+ util.tableEnv.getConfig.getConfiguration.setBoolean(TABLE_EXEC_MINIBATCH_ENABLED, true)
+ util.tableEnv.getConfig.getConfiguration.setLong(TABLE_EXEC_MINIBATCH_SIZE, 3L)
+ util.tableEnv.getConfig.getConfiguration.set(
+ TABLE_EXEC_MINIBATCH_ALLOW_LATENCY, Duration.ofSeconds(1))
+ val sql =
+ """
+ |SELECT COUNT(b) FROM (
+ | SELECT a, b, c
+ | FROM (
+ | SELECT *,
+ | ROW_NUMBER() OVER (PARTITION BY a ORDER BY rowtime DESC) as rank_num
+ | FROM MyTable)
Review comment:
`MyTable` is view from DataStream. Would be great if we can add test for DDL with watermark to verify `MiniBatchAssigner` also works well with `WatermarkAssigner`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r555537296
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
hah, you're right @beyond1920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73247f65a10d8a1e4df8608044c7c9c811505520 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r556225680
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
Okay, I'm ok with this change, that's better if you could add a test for lastRow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758022239
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 0482293fe8ca9fdd51defaa6db8e86b35dac1c5f (Fri May 28 07:09:12 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on a change in pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on a change in pull request #14610:
URL: https://github.com/apache/flink/pull/14610#discussion_r555537296
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/DeduplicateITCase.scala
##########
@@ -226,6 +226,42 @@ class DeduplicateITCase(miniBatch: MiniBatchMode, mode: StateBackendMode)
assertEquals(expected.sorted, rawResult.sorted)
}
+ @Test
+ def testLastRowOnRowTimeFollowedByUnboundedAgg(): Unit = {
Review comment:
hah, you're right @beyond1920 , So the test was added to test the output of `Deduplicate` can be consumed by unbounded `agg`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
}, {
"hash" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123",
"triggerID" : "384aa1d095bc02c376ec2a0af43501a19e2a81ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384aa1d095bc02c376ec2a0af43501a19e2a81ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12123)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758022239
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 73247f65a10d8a1e4df8608044c7c9c811505520 (Mon Jan 11 15:25:50 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9f9cf0caee79ac6382288b98ec98cad50a706cc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-758033846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890",
"triggerID" : "73247f65a10d8a1e4df8608044c7c9c811505520",
"triggerType" : "PUSH"
}, {
"hash" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067",
"triggerID" : "c9f9cf0caee79ac6382288b98ec98cad50a706cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73247f65a10d8a1e4df8608044c7c9c811505520 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11890)
* c9f9cf0caee79ac6382288b98ec98cad50a706cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12067)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #14610: [FLINK-20909][table-planner-blink] Fix deduplicate mini-batch interval infer
Posted by GitBox <gi...@apache.org>.
leonardBang commented on pull request #14610:
URL: https://github.com/apache/flink/pull/14610#issuecomment-760677892
LGTM cc @wuchong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org