[GitHub] [pulsar] phemmer opened a new issue #6866: Go function support for manual consumer

[GitBox <gi...@apache.org>]

phemmer opened a new issue #6866:
URL: https://github.com/apache/pulsar/issues/6866


   **Is your feature request related to a problem? Please describe.**
   Right now the only way to use go functions is to have pulsar-function-go setup the consumers & producers for you, where it invokes your handler when a message comes in. I've been struggling with numerous difficulties due to this design limitation.
   
   1. In my application, it would make the design much simpler and cleaner if I could read a message from the input topic when I'm ready instead of having my handler called.  The latter requires restructuring of the application which makes it difficult to understand/manage.
   2. In some scenarios, I want to read a few messages off the queue without having to acknowledge one before obtaining the next. Basically I want to perform a "peek" operation. This cannot be done with the handler design.
   3. In another scenario, where there are multiple input topics, I want to pause reading from some topics, and only read from others for a little while.
   5. These things could be implemented manually, by instantiating the consumers without using pulsar-function-go, however then you lose things like the automatic setup of the logger, health checks, etc.
   
   Also in a related case, if I want to use pulsar functions to effectively construct a custom source (integrate with an external system and not have any input topics), this currently cannot be done. This may be an abuse of functions, but currently true custom sources can only be implemented in Java.
   
   Also related is if I want my function to perform some initialization tasks, which require the function context (e.g. retrieving data from state storage), I basically have to wait for the first message to come in to be able to do so.
   
   **Describe the solution you'd like**
   All in all, I think all the above described issues could be addressed if the function instance & context objects could be created without specifying a handler, and were then exposed to the user after setup. Then the user could use the consumers & producers directly.
   
   **Describe alternatives you've considered**
   The only alternative right now is to not use the pulsar-function-go package, but to set everything up manually. However then you lose all the other things that pulsar-function-go provides (logger, health checks, etc), and have to set those up manually as well.
   
   **Additional context**
   I think that if this were addressed, it would provide a solution for #5137 as well.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org