You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/10/09 16:03:29 UTC
[GitHub] [incubator-doris] morningman opened a new issue #6803: [Optimize] Avoid too many abort task in routine load job
morningman opened a new issue #6803:
URL: https://github.com/apache/incubator-doris/issues/6803
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Currently, even if there is no more data in kafka, the routine load job will still send task to try to
consume from the kafka topic. And you will see a lot of `kafka consume timeout` in be.INFO.
And finally this task is aborted, and the `abortedTaskNum` of routine load will increase by 1.
So I think we can try to fetch the latest offset of kafka partition before deciding whether to send task
to consume. This will bring 2 benefits:
1. Avoid unnecessary tasks running on BE, which occupying working thread.
2. Avoid lots of aborted transactions.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed issue #6803: [Optimize] Avoid too many abort task in routine load job
Posted by GitBox <gi...@apache.org>.
yangzhg closed issue #6803:
URL: https://github.com/apache/incubator-doris/issues/6803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed issue #6803: [Optimize] Avoid too many abort task in routine load job
Posted by GitBox <gi...@apache.org>.
yangzhg closed issue #6803:
URL: https://github.com/apache/incubator-doris/issues/6803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org