You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "learner1212 (via GitHub)" <gi...@apache.org> on 2023/04/11 02:12:08 UTC
[GitHub] [doris] learner1212 opened a new issue, #18535: [Feature] Routine load process with a exist group id
learner1212 opened a new issue, #18535:
URL: https://github.com/apache/doris/issues/18535
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
routine load now not support modify columns, so when my table schema changed, we must stop current routine load job and submit a new one. but doris will reset offset to lastest or earilest by property.kafka_default_offsets setting, that may cause data repeat or data lost. the only way we avoid this problem is appoint kafka_partitions and kafka_offsets settings, and this also bring new problems:
1. the configuration is difficult when we have a large number of partitions;
2. when we increase partiton in topic, we will loss data in the new partitions.
in summary， we need to consume kafka topic with a exist consumer group an do not reset the offset.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] reswqa commented on issue #18535: [Feature] Routine load process with a exist group id
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on issue #18535:
URL: https://github.com/apache/doris/issues/18535#issuecomment-1507837596
Thanks for reporting this, I'd like to help support this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on issue #18535: [Feature] Routine load process with a exist group id
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on issue #18535:
URL: https://github.com/apache/doris/issues/18535#issuecomment-1502936379
Sounds reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] learner1212 commented on issue #18535: [Feature] Routine load process with a exist group id
Posted by "learner1212 (via GitHub)" <gi...@apache.org>.
learner1212 commented on issue #18535:
URL: https://github.com/apache/doris/issues/18535#issuecomment-1507812619
> Can we support this feature in future versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] learner1212 commented on issue #18535: [Feature] Routine load process with a exist group id
Posted by "learner1212 (via GitHub)" <gi...@apache.org>.
learner1212 commented on issue #18535:
URL: https://github.com/apache/doris/issues/18535#issuecomment-1507812861
> Sounds reasonable
Can we support this feature in future versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on issue #18535: [Feature] Routine load process with a exist group id
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on issue #18535:
URL: https://github.com/apache/doris/issues/18535#issuecomment-1507820347
yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org