You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/04 13:50:51 UTC
[GitHub] [pulsar] sbourkeostk opened a new pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
sbourkeostk opened a new pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482
Fixes #9480
### Motivation
The kafka source sets auto.offset.reset to "earliest". This means all old messages from kafka are produced to pulsar. Often is it desirable to start form the present location "latest".
The option is set after the user config has been loaded so it cannot be changed:
[source code link](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L87)
### Modifications
Added an autoOffsetReset option to KafkaSourceConfig
### Verifying this change
- [x] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change added tests and can be verified as follows:
Added to unit test and verified behaviour.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): no
- The public API: no
- The schema: no
- The default values of configurations: no
- The wire protocol: no
- The rest endpoints: no
- The admin cli options: no
- Anything that affects deployment: no
### Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs / JavaDocs
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#discussion_r570676678
##########
File path: site2/docs/adaptors-kafka.md
##########
@@ -128,7 +128,7 @@ Properties:
| Config property | Supported | Notes |
|:----------------------------------------|:----------|:------------------------------------------------------------------------------|
| `acks` | Ignored | Durability and quorum writes are configured at the namespace level |
-| `auto.offset.reset` | Yes | Will have a default value of `latest` if user does not give specific setting. |
+| `auto.offset.reset` | Yes | Will have a default value of `earliest` if user does not give specific setting. |
Review comment:
```suggestion
| `auto.offset.reset` | Yes | It uses a default value of `earliest` if you do not give a specific setting. |
```
In technical information, follow these general guidelines for personal pronouns:
• Avoid the first-person pronouns I and we, except in these situations:
° In the question portion of frequently asked questions (FAQs)
° In articles, white papers, or documents that have listed authors and in which the authors describe their own actions or opinions
• Use the second-person pronoun you as much as possible. The subject of an imperative sentence is understood to be you.
• Avoid third-person pronouns that are gender specific.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#discussion_r570676678
##########
File path: site2/docs/adaptors-kafka.md
##########
@@ -128,7 +128,7 @@ Properties:
| Config property | Supported | Notes |
|:----------------------------------------|:----------|:------------------------------------------------------------------------------|
| `acks` | Ignored | Durability and quorum writes are configured at the namespace level |
-| `auto.offset.reset` | Yes | Will have a default value of `latest` if user does not give specific setting. |
+| `auto.offset.reset` | Yes | Will have a default value of `earliest` if user does not give specific setting. |
Review comment:
```suggestion
| `auto.offset.reset` | Yes | It uses a default value of `earliest` if you do not give a specific setting. |
```
In technical information, follow these general guidelines for personal pronouns:
• Avoid the first-person pronouns I and we, except in these situations:
° In the question portion of frequently asked questions (FAQs)
° In articles, white papers, or documents that have listed authors and in which the authors describe their own actions or opinions
• Use the second-person pronoun you as much as possible. The subject of an imperative sentence is understood to be you.
• Avoid third-person pronouns that are gender specific.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-778092361
/pulsarbot rerun-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-778487698
/pulsarbot rerun-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-774032694
/pulsarbot rerun-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
Renkai commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-773373578
@BewareMyPower Could you have a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on a change in pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on a change in pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#discussion_r570238308
##########
File path: site2/docs/adaptors-kafka.md
##########
@@ -128,7 +128,7 @@ Properties:
| Config property | Supported | Notes |
|:----------------------------------------|:----------|:------------------------------------------------------------------------------|
| `acks` | Ignored | Durability and quorum writes are configured at the namespace level |
-| `auto.offset.reset` | Yes | Will have a default value of `latest` if user does not give specific setting. |
Review comment:
This is a correction to an error in the existing docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-773445044
Sample config added to kafkaSourceConfig.yaml and verified in loadFromYamlFileTest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Renkai commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
Renkai commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-773373578
@BewareMyPower Could you have a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-775011825
/pulsarbot rerun-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on a change in pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on a change in pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#discussion_r570238308
##########
File path: site2/docs/adaptors-kafka.md
##########
@@ -128,7 +128,7 @@ Properties:
| Config property | Supported | Notes |
|:----------------------------------------|:----------|:------------------------------------------------------------------------------|
| `acks` | Ignored | Durability and quorum writes are configured at the namespace level |
-| `auto.offset.reset` | Yes | Will have a default value of `latest` if user does not give specific setting. |
Review comment:
This is a correction to an error in the existing docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sbourkeostk commented on pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sbourkeostk commented on pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482#issuecomment-773445044
Sample config added to kafkaSourceConfig.yaml and verified in loadFromYamlFileTest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #9482: [Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #9482:
URL: https://github.com/apache/pulsar/pull/9482
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org