You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zxealous (via GitHub)" <gi...@apache.org> on 2023/04/28 08:37:38 UTC
[GitHub] [doris] zxealous opened a new issue, #19204: [Bug] Broker load task slows down
zxealous opened a new issue, #19204:
URL: https://github.com/apache/doris/issues/19204
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.2, master
### What's Wrong?
Broker load slows down and importing 2G data takes nearly 3 hours.
Load process:
In func PlanFragmentExecutor::open_vectorized_internal(),
Single threaded loop usage function `get_vectorized_internal()` read data and send by `_sink->send()`, `_sink->send()` will call `VNodeChannel::add_block`. In `VNodeChannel::add_block`, if `_pending_batches_bytes > _max_pending_batches_bytes` the thread will be always sleep and do nothing, until `_pending_batches_bytes` <= `_max_pending_batches_bytes`, So, if one NodeChannel's `_pending_batches_bytes` is full, the thread will always sleep, even if other NodeChannel's `_pending_batches_bytes` is not full.
`_pending_batches_bytes` is full because of the method PInternalServiceImpl::tablet_writer_add_block() has not completed transmitting data, and we found the reason of transmit data need too long time is MemTable::shrink_memtable_by_agg() called in DeltaWriter::write is time-consuming, MemTable::shrink_memtable_by_agg() is synchronous.
### What You Expected?
work well
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] chenlinzhong closed issue #19204: [Bug] shrink_memtable_by_agg causes all data imports to be very slow
Posted by "chenlinzhong (via GitHub)" <gi...@apache.org>.
chenlinzhong closed issue #19204: [Bug] shrink_memtable_by_agg causes all data imports to be very slow
URL: https://github.com/apache/doris/issues/19204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org