[GitHub] [pulsar] Lanayx opened a new issue #6531: Add subscriptionPosition as parameters on function createion

[GitBox <gi...@apache.org>]

Lanayx opened a new issue #6531: Add subscriptionPosition as parameters on function createion
URL: https://github.com/apache/pulsar/issues/6531
 
 
   **Is your feature request related to a problem? Please describe.**
   I'd like to use pulsar functions for request-response pattern where topics are dynamically created per application node. For that I'd like to use pattern-based function, but it is unable to grab initial messages that got into the topic before pattern-based function recognizes them.
   
   **Describe the solution you'd like**
   I'd like to have subscriptionPosition as a parameter, to set it to earliest, so no messages will are lost
   
   **Describe alternatives you've considered**
   There is a workaround of manual topics creation before function creation, but this is complicated from application point of view, it's much easier to have topics created automatically
   
   **Additional context**
   Slack conversation:
   ```
   Lanayx  1:07 AM
   Looks like there is an issue regarding pattern-topic function - it appears that first message that creates the topic is lost, even if topic has retention, because there is no way to set subscriptionPosition to earliest for function. There is a workaround for single topic with create-subscription command, but this command doesn't support regex for topics. Am I correct that this is a bug?
   
   David K  1:09 AM
   Did you create the consuming function first or the message producer first?  If you create the function first, then it should receive all messages after it subscribes.
   
   Lanayx  1:10 AM
   I created the consuming function first, but since it's regex-based function it doesn't create topic, the topic is created from producer side together with first message, so once function recognizes the new topic existence it subscribes, but is unable to grab first message. By the way I've just realized that not only the first, but all messages will be lost until function subscribes to the new topic (edited) 
   
   David K  1:14 AM
   That is the expected behavior based on what you have told me. However, you can create the topic first (manually) and then the function to get the desired behavior.
   
   Lanayx  1:18 AM
   It's expected but not desired :slightly_smiling_face: A very common case for RabbitMQ-based apps is to create topics dynamically from the application to implement Request-Response pattern, and have some function recognize them. Looks like subscriptionPosition is an important setting to have for functions to handle this. (edited) 
   
   David K  1:19 AM
   Sounds like a feature request.  In Pulsar topics have to exist first, then subscriptions (which track where you are in reading from in the topic) get created as they are associated with the topic.
   1:20
   Functions use the Consumer interface underneath the covers, which supports subscription-based consumption. What you need is Reader based consumption which allows you to specify the starting point of your reads.  :smiley:
   1:21
   So maybe a PIP for "Support Reader interface based Functions" would be the way to go?
   
   Lanayx  1:25 AM
   I don't think I need a reader, pattern-based function is already a solution, I plan to create input-output request-like topic for each new application node dynamically and it works well except of first message that is lost due to lack of subscriptionPosition=earliest
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services