You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/21 06:46:43 UTC
[GitHub] [flink] tweise opened a new pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
tweise opened a new pull request #14711:
URL: https://github.com/apache/flink/pull/14711
## What is the purpose of the change
*Ensure that KafkaSourceEnumerator honors the user provided consumer properties*
## Verifying this change
I tested this change on an internal Kafka cluster, we should probably also add unit test coverage. Putting this up for discussion.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
}, {
"hash" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 745be8be878cbc0eb4094daa66ecce499d5ac585 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318)
* 77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-766512749
@becketqin unit test was added, PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 745be8be878cbc0eb4094daa66ecce499d5ac585 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 745be8be878cbc0eb4094daa66ecce499d5ac585 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
becketqin commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-767418290
One thing I am not sure is that whether the usage of `WhiteBox` is a common pattern for unit tests. It seems it is only used occasionally. Other unit tests seem using alternatives such as package private methods. @zentol curious what is your opinion about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-768422261
@becketqin thanks for the review.
In this case `WhiteBox` is needed to check that the settings are reflected in the Kafka client objects. Should there be ideas how to solve this in a better way I will be happy to take it as follow-up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764423225
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise merged pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
tweise merged pull request #14711:
URL: https://github.com/apache/flink/pull/14711
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] becketqin commented on a change in pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
becketqin commented on a change in pull request #14711:
URL: https://github.com/apache/flink/pull/14711#discussion_r562419345
##########
File path: flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java
##########
@@ -309,7 +309,8 @@ private void assignPendingPartitionSplits() {
private AdminClient getKafkaAdminClient() {
Properties adminClientProps = new Properties();
- copyProperty(properties, adminClientProps, ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
+ adminClientProps.putAll(properties);
Review comment:
It seems we also need to set the user properties in `getKafkaConsumer()` as well in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764423225
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc (Fri May 28 08:14:36 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764423225
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 93b4ec3111f193bdeb091c11100ecf916389527e (Thu Jan 21 06:49:49 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
becketqin commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-767410732
Thanks for the patch. LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
}, {
"hash" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12435",
"triggerID" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12435)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 745be8be878cbc0eb4094daa66ecce499d5ac585 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-766512749
@becketqin unit test was added, PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14711:
URL: https://github.com/apache/flink/pull/14711#issuecomment-764431519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318",
"triggerID" : "745be8be878cbc0eb4094daa66ecce499d5ac585",
"triggerType" : "PUSH"
}, {
"hash" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12435",
"triggerID" : "77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 745be8be878cbc0eb4094daa66ecce499d5ac585 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12318)
* 77a21ef4ac1f3f1cdade8beb3174a5e89a3d3bfc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12435)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org