[I] [Bug] Sink inputSpecs schemaProperties is silently dropped by SinkConfigUtils in both conversion directions [pulsar]

["david-streamlio (via GitHub)" <gi...@apache.org>]

david-streamlio opened a new issue, #26425:
URL: https://github.com/apache/pulsar/issues/26425

   **Search before asking**
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   ### Version
   
   Reproduced on `master` (`e4d106f6ee`). The same omission is present on the 4.0.x line.
   
   ### Minimal reproduce step
   
   Create a sink whose `inputSpecs` entry carries `schemaProperties`:
   
   ```json
   {
     "tenant": "public",
     "namespace": "default",
     "name": "test-sink",
     "inputSpecs": {
       "persistent://public/default/in": {
         "schemaType": "avro",
         "schemaProperties": { "__alwaysAllowNull": "true" },
         "consumerProperties": { "application": "billing" }
       }
     }
   }
   ```
   
   Then read it back:
   
   ```
   pulsar-admin sinks get --tenant public --namespace default --name test-sink
   ```
   
   ### What did you expect to see?
   
   `schemaProperties` preserved on the round trip, the same way `consumerProperties` is, and the same way it already works for Pulsar Functions.
   
   ### What did you see instead?
   
   `schemaProperties` is silently dropped. The API accepts the value, but it never reaches `ConsumerSpec` and is absent from the response. There is no error or warning, so the caller has no signal that the setting was discarded.
   
   ### Anything else?
   
   The proto field exists — `Function.proto:111` defines `map<string, string> schemaProperties = 5` on `ConsumerSpec` — and `ConsumerConfig.schemaProperties` exists on the model. Only the **sink** conversion path fails to use them, in both directions:
   
   - [`SinkConfigUtils.convert()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L148-L171) builds each `ConsumerSpec` from the `inputSpecs` entry and copies `schemaType`/`serdeClassName`, `receiverQueueSize`, `cryptoSpec`, `messagePayloadProcessorSpec`, `consumerProperties` and `poolMessages` — but never `schemaProperties`.
   - [`SinkConfigUtils.convertFromDetails()`](https://github.com/apache/pulsar/blob/master/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java#L290-L313) restores the same set on the way back, and likewise never restores `schemaProperties`.
   
   `FunctionConfigUtils` handles the field correctly in both directions (`putSchemaProperties` at lines 137/161/248, `forEachSchemaProperties` at line 432), which is what makes this sink-specific rather than a general `ConsumerConfig` limitation.
   
   **Practical impact.** Any client that exposes `schemaProperties` on a sink input spec exposes a write-only field: the value is accepted, dropped, and then reads back absent, producing permanent drift for declarative tooling. This surfaced while reviewing a Terraform provider change (streamnative/terraform-provider-pulsar#224), where exposing the attribute had to be deferred for exactly this reason.
   
   Suggested fix: copy `schemaProperties` in both `convert()` and `convertFromDetails()`, mirroring the `consumerProperties` handling, and add round-trip coverage in `SinkConfigUtilsTest`.
   
   I am happy to put up a PR for this.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org