[I] [Go Functions] Most of ConsumerSpec is ignored: cryptoSpec, schemaProperties, consumerProperties, poolMessages [pulsar]

["david-streamlio (via GitHub)" <gi...@apache.org>]

david-streamlio opened a new issue, #26407:
URL: https://github.com/apache/pulsar/issues/26407

   Master Issue: #26404
   
   ### Search before reporting
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   ### Motivation
   
   `ConsumerSpec` carries nine fields. The Go runtime reads two of them.
   
   ```protobuf
   message ConsumerSpec {
       string schemaType = 1;                                  // read
       string serdeClassName = 2;                              // read (schema selection)
       bool isRegexPattern = 3;                                // read
       ReceiverQueueSize receiverQueueSize = 4;                // read
       map<string, string> schemaProperties = 5;               // NOT read
       map<string, string> consumerProperties = 6;             // NOT read
       CryptoSpec cryptoSpec = 7;                              // NOT read
       bool poolMessages = 8;                                  // NOT read
       MessagePayloadProcessorSpec messagePayloadProcessorSpec = 9;  // NOT read
   }
   ```
   
   `git grep -in "CryptoSpec\|SchemaProperties\|ConsumerProperties\|PoolMessages\|MessagePayloadProcessor" -- 'pulsar-function-go/pf/*.go'` returns nothing.
   
   The Python runtime applies `schemaProperties` and `cryptoSpec` (`python_instance.py` builds a crypto key reader through `get_crypto_reader(consumer_conf.cryptoSpec)` and threads `schemaProperties` into `get_schema`), and the Java runtime applies all of them.
   
   `cryptoSpec` is the one with real consequences: a Go function consuming from an encrypted topic gets no crypto key reader, so it cannot decrypt. The rest are quieter — `consumerProperties` are invisible to broker-side tooling that reads them, `poolMessages` leaves a performance option unavailable, and `schemaProperties` are dropped from schema resolution.
   
   ### Solution
   
   Apply the remaining `ConsumerSpec` fields in `setupConsumer`, mapping onto `pulsar.ConsumerOptions`. The Go client has an equivalent for most of them:
   
   | `ConsumerSpec` field | `pulsar.ConsumerOptions` |
   | --- | --- |
   | `schemaProperties` | schema properties on the resolved `Schema` |
   | `consumerProperties` | `Properties` |
   | `cryptoSpec` | `Decryption` (`*MessageDecryptionInfo`) |
   | `poolMessages` | no direct equivalent — needs confirming |
   | `messagePayloadProcessorSpec` | no equivalent — likely out of scope |
   
   Grouped into one issue because a single change to `setupConsumer` covers them, but they are separable and a partial fix is useful. Suggested order by impact: `cryptoSpec`, then `consumerProperties` and `schemaProperties`, then the two that may have no Go client equivalent.
   
   Where a field genuinely cannot be supported, refusing it explicitly at startup — the way `EFFECTIVELY_ONCE` is refused at `instanceConf.go:137` — would be better than continuing to ignore it. That is the point made in the master issue.
   
   ### Alternatives
   
   Leaving these unimplemented is defensible for `messagePayloadProcessorSpec`, which is a Java-centric extension point. It is not defensible for `cryptoSpec`, where the effect is that an encrypted topic cannot be consumed at all.
   
   ### Anything else?
   
   Verified against `origin/master`.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org