[GitHub] [pulsar] devinbost opened a new issue #9986: [Functions] Need a way to disable message batching between functions

[GitBox <gi...@apache.org>]

devinbost opened a new issue #9986:
URL: https://github.com/apache/pulsar/issues/9986


   **Is your feature request related to a problem? Please describe.**
   It has been identified that there is a major stability issue that appears to be associated with message batching. As Pulsar functions currently always have `.enableBatching(true)` set during instance creation, there needs to be a way to disable message batching per user config. 
   
   **Describe the solution you'd like**
   I'm proposing the ability to set a default function batching setting in WorkerConfig that will affect the entire cluster. Then, if individual functions need to be set more specifically during deployment, a setting in their FunctionDetails will allow that customization as well. 
   
   **Describe alternatives you've considered**
   Although an ideal PR would fix the underlying stability issues with message batching, we need a working solution ASAP for high-traffic Pulsar function-heavy workloads, so allowing users to disable message batching will give them that flexibility in case they are having issues with it in their function workloads. This PR should get those users to a working state quickly and buy more time to create a fix for the underlying issues with message batching. 
   
   **Additional context**
   In the event that message batching actually isn't the cause of the freezing topic issue, well, this will still be a nice feature and will help us rule that out. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org