[GitHub] rdhabalia commented on a change in pull request #1903: Support pulsar source to start consumer with topic patterns

[GitBox <gi...@apache.org>]

rdhabalia commented on a change in pull request #1903: Support pulsar source to start consumer with topic patterns
URL: https://github.com/apache/incubator-pulsar/pull/1903#discussion_r192929309
 
 

 ##########
 File path: pulsar-functions/proto/src/main/proto/Function.proto
 ##########
 @@ -70,6 +70,7 @@ message SourceSpec {
     SubscriptionType subscriptionType = 3;
     map<string,string> topicsToSerDeClassName = 4;
     uint64 timeoutMs = 6;
+    string topicsPattern = 7;
 
 Review comment:
   I thought about it earlier but then I feel we can combine it in `topicsToSerDeClassName` only. in one of the usecase, topicPattern can be used to start all consumers under a given namespace. now, there is a possibility that some of the topics may need different deserializers so, PulsarSource can first check deserializer for a specific topic in the map and then try to match the pattern in the same map. in this way, we can save cpu by avoiding pattern-matching on every message and will be easier for user to configure it in function-config `topicToSerDeMap.put("pattern", deserializer)`
   
   ```
   +            SerDe deserializer = null;
   +            if (this.topicToSerDeMap.containsKey(topicName)) {
   +                deserializer = this.topicToSerDeMap.get(topicName);
   +            } else if (isTopicsPattern) {
   +                deserializer = this.topicToSerDeMap.get(this.pulsarSourceConfig.getTopicsPattern());
   +            }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services