[GitHub] [incubator-doris] morningman opened a new issue #2181: Too many transaction waiting to be cleared on BE

[GitBox <gi...@apache.org>]

morningman opened a new issue #2181: Too many transaction waiting to be cleared on BE
URL: https://github.com/apache/incubator-doris/issues/2181
 
 
   **Describe the bug**
   When a user submits an load transaction, the load transaction may succeed on some BE nodes and some BE nodes fail. Eventually, this load transaction will be considered a failure by the FE node. After that, the remaining transactions on those successful nodes will be cleaned up. The cleanup is done after BE node report its own transaction status to the FE, and the FE node decides whether to send the cleanup task.
   
   When a user submits a large number of load transactions in a short period of time, and these transactions happen to succeed on some nodes, some nodes fail. The successful node will accumulate a large number of transactions to be cleaned up. A large number of transactions will cause the report to run very slowly. Eventually the node is stuck.
   
   **How to fix**
   When an load transaction fails, we should actively send the cleanup task instead of relying entirely on the BE node's report.
   
   At the same time, the BE node should have a negative feedback mechanism, that is, when there are too many load transactions, reject the load transaction request that arrives later.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org