You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/10/27 23:15:23 UTC
[GitHub] [flink] john-karp opened a new pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
john-karp opened a new pull request #17583:
URL: https://github.com/apache/flink/pull/17583
## What is the purpose of the change
Add UniformShardAssigner to assign Kinesis shards to subtasks. This algorithm takes advantage of the Kinesis hash-key space that each shard occupies. By default, each shard is assigned an equal-sized range of hash-key space. And UpdateShardCount (with `UNIFORM_SCALING`) tries to preserve this. So we can nicely assign shards to subtasks by mapping through hash-key space.
However, some users may have customized how shards are distributed, such that they occupy ranges of hash-key space that vary greatly in size. For such users, the default shard assignment will likely work better. Hence, I have not changed the default here.
Please refer to the Jira ticket for a longer write-up, including performance comparison with the default assigner.
## Brief change log
- Add UniformShardAssigner
- Unit testing for above
- Documentation for above
## Verifying this change
This change added tests and can be verified as follows:
- Added unit test to ensure hash key space is correctly mapped to subtask-index space.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - new class with @Public annotation
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs + JavaDocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] CrynetLogistics commented on a change in pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
CrynetLogistics commented on a change in pull request #17583:
URL: https://github.com/apache/flink/pull/17583#discussion_r744733412
##########
File path: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/util/UniformShardAssignerTest.java
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis.util;
+
+import org.apache.flink.streaming.connectors.kinesis.model.StreamShardHandle;
+import org.apache.flink.util.TestLogger;
+
+import com.amazonaws.services.kinesis.model.HashKeyRange;
+import com.amazonaws.services.kinesis.model.Shard;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.math.BigInteger;
+import java.util.stream.Stream;
+
+/** Tests for the {@link UniformShardAssigner}. */
+public class UniformShardAssignerTest extends TestLogger {
Review comment:
I like how you've added the stream specific offset to seed the distribution to different streams. But I think, you need to take the modulo in your tests just like the caller of `assign()` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953379268
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 7b0498e95e6afcdcac2b448455c76cad101c00fb (Wed Oct 27 23:19:03 UTC 2021)
**Warnings:**
* Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket.
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24639).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
* 4e74b80beca567145c62de1005fe4b527eb2f2ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b0498e95e6afcdcac2b448455c76cad101c00fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b0498e95e6afcdcac2b448455c76cad101c00fb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b0498e95e6afcdcac2b448455c76cad101c00fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547)
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e74b80beca567145c62de1005fe4b527eb2f2ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-957322519
@dannycranmer @CrynetLogistics can one of you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
* 4e74b80beca567145c62de1005fe4b527eb2f2ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dannycranmer merged pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
dannycranmer merged pull request #17583:
URL: https://github.com/apache/flink/pull/17583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b0498e95e6afcdcac2b448455c76cad101c00fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-957322519
@dannycranmer @CrynetLogistics can one of you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #17583:
URL: https://github.com/apache/flink/pull/17583#discussion_r744866155
##########
File path: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/util/UniformShardAssignerTest.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis.util;
+
+import org.apache.flink.streaming.connectors.kinesis.model.StreamShardHandle;
+import org.apache.flink.util.TestLogger;
+
+import com.amazonaws.services.kinesis.model.HashKeyRange;
+import com.amazonaws.services.kinesis.model.Shard;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.math.BigInteger;
+import java.util.stream.Stream;
+
+/** Tests for the {@link UniformShardAssigner}. */
Review comment:
I am afraid the `TestLogger` does not work here. It is built for junit4. Please use `TestLoggerExtension` with the `@ExtendWith` annotation.
##########
File path: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/UniformShardAssigner.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis.util;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.streaming.connectors.kinesis.KinesisShardAssigner;
+import org.apache.flink.streaming.connectors.kinesis.model.StreamShardHandle;
+
+import com.amazonaws.services.kinesis.model.HashKeyRange;
+
+import java.math.BigInteger;
+
+/**
+ * A {@link KinesisShardAssigner} that maps Kinesis shard hash-key ranges to Flink subtasks. It
+ * creates a more uniform distribution of shards across subtasks than {@link
+ * org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher#DEFAULT_SHARD_ASSIGNER}
+ * when the Kinesis records in the stream have hash keys that are uniformly distributed over all
+ * possible hash keys, which is the case if records have randomly-generated partition keys. (This is
+ * the same assumption made if you use the Kinesis UpdateShardCount operation with UNIFORM_SCALING.)
+ */
+@Public
Review comment:
Nit: The interface you are implementing is `@PublicEvolving` but this class `@Public` was this intentionally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dannycranmer commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-965527617
Thanks @john-karp, LGTM, merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
* 4e74b80beca567145c62de1005fe4b527eb2f2ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26183)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dannycranmer commented on a change in pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17583:
URL: https://github.com/apache/flink/pull/17583#discussion_r744921228
##########
File path: docs/content/docs/connectors/datastream/kinesis.md
##########
@@ -121,10 +121,13 @@ then some consumer subtasks will simply be idle and wait until it gets assigned
new shards (i.e., when the streams are resharded to increase the
number of shards for higher provisioned Kinesis service throughput).
-Also note that the assignment of shards to subtasks may not be optimal when
-shard IDs are not consecutive (as result of dynamic re-sharding in Kinesis).
-For cases where skew in the assignment leads to significant imbalanced consumption,
-a custom implementation of `KinesisShardAssigner` can be set on the consumer.
+Also note that the default assignment of shards to subtasks is based on the hashes of the shard and stream names,
+which will more-or-less balance the shards across the subtasks.
+However, assuming the default Kinesis shard management is used on the stream (UpdateShardCount with `UNIFORM_SCALING`),
+setting `UniformShardAssigner` as the shard assigner on the consumer will much more evenly distribute shards to subtasks.
+Assuming the incoming Kinesis records are assigned random Kinesis `PartitionKey` or `ExplicitHashKey` values,
+the result is consistent subtask loading.
+If neither the default assigner nor the `UniformShardAssigner` suffice, a custom implementation of `KinesisShardAssigner` can be set.
Review comment:
Can you please also update the CN docs too so that they stay in sync?
- https://github.com/apache/flink/blob/master/docs/content.zh/docs/connectors/table/kinesis.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dannycranmer commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-963368030
Looks good to me, besides one comment about `cn` comments. Please rebase to resolve conflicts and I will merge one CI passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
* 4e74b80beca567145c62de1005fe4b527eb2f2ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
}, {
"hash" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179",
"triggerID" : "3fff24e99824ed17628bad05d9f7a7f946cd6ed8",
"triggerType" : "PUSH"
}, {
"hash" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e74b80beca567145c62de1005fe4b527eb2f2ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
* 3fff24e99824ed17628bad05d9f7a7f946cd6ed8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26179)
* 4e74b80beca567145c62de1005fe4b527eb2f2ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b0498e95e6afcdcac2b448455c76cad101c00fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547)
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-953378438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25547",
"triggerID" : "7b0498e95e6afcdcac2b448455c76cad101c00fb",
"triggerType" : "PUSH"
}, {
"hash" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599",
"triggerID" : "308e3f7d2ce15ce4434dddccbb4f53350e1f081a",
"triggerType" : "PUSH"
}, {
"hash" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170",
"triggerID" : "293fa0d578b8ab4f4ef71bd19e4d968111dd65d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 308e3f7d2ce15ce4434dddccbb4f53350e1f081a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25599)
* 293fa0d578b8ab4f4ef71bd19e4d968111dd65d8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26170)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
fapaul commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-957322519
@dannycranmer @CrynetLogistics can one of you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] CrynetLogistics commented on pull request #17583: [FLINK-24639][kinesis] Add UniformShardAssigner
Posted by GitBox <gi...@apache.org>.
CrynetLogistics commented on pull request #17583:
URL: https://github.com/apache/flink/pull/17583#issuecomment-963298049
Great! Looks good to me. Thanks @john-karp. Was wondering if @dannycranmer had any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org