You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "david-streamlio (via GitHub)" <gi...@apache.org> on 2026/08/25 15:59:07 UTC
[PR] [improve][doc] Document producerConfig.batchingConfig for Pulsar Functions [pulsar-site]
david-streamlio opened a new pull request, #1215:
URL: https://github.com/apache/pulsar-site/pull/1215
### Motivation
[PIP-401](https://github.com/apache/pulsar/blob/master/pip/pip-401.md) added a `batchingConfig` field to a function's `producerConfig`, letting a function enable or disable batching and tune the batch size, byte limit, publish delay, and batch builder.
**None of it is documented.** `batchingConfig` appears nowhere in the docs, and the `ProducerConfig` table in `functions-cli.md` lists every sibling field — `maxPendingMessages`, `cryptoConfig`, `batchBuilder`, `compressionType` — but not this one. All six sub-fields are invisible to anyone who has not read `BatchingConfig.java`.
### Modifications
**Add a `BatchingConfig` section** covering the six fields with their defaults (`enabled` = `true`, `batchingMaxPublishDelayMs` = `10`), and a `batchingConfig` row linking to it from the `ProducerConfig` table — the pattern the table already uses for `cryptoConfig` → [CryptoConfig](#cryptoconfig).
**Record that only Java applies it today.** The Python and Go runtimes hard-code batching to enabled with a 10 ms maximum publish delay and ignore `batchingConfig` entirely, tracked as apache/pulsar#26390 and apache/pulsar#26391. Documenting the fields without saying so would just move the surprise rather than remove it: the configuration is accepted, stored, and echoed back by `functions get`, and then ignored.
Two further notes on rows that were **already** in the table, both describing current behaviour:
- **`batchingConfig.batchBuilder` takes precedence over `producerConfig.batchBuilder`**, matching the Java runtime. A user who sets both and sees one ignored has no way to find that out today.
- **The Go client does not implement `SNAPPY`.** `instance.go` maps it to `LZ4` in the default arm of its compression switch, so a Go function configured with `SNAPPY` silently gets `LZ4`. This one is independent of the rest of the PR and easy to drop if you would rather it went separately.
### Versioned docs
`docs/` only. `batchingConfig` exists in released versions, so a case could be made for back-porting the section; I have kept this to `docs/` to stay reviewable, and am happy to add versioned copies if that is preferred.
### Follow-up
apache/pulsar#26392 and apache/pulsar#26393 make the Python and Go runtimes honour these settings. When those land, the note in this section needs to become a smaller one — `roundRobinRouterBatchingPartitionSwitchFrequency` stays Java-only, and `maxPendingMessagesAcrossPartitions` has no Go equivalent. I will open that as a separate PR rather than describing behaviour that has not shipped.
### Documentation
This *is* the documentation change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][doc] Document producerConfig.batchingConfig for Pulsar Functions [pulsar-site]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1215:
URL: https://github.com/apache/pulsar-site/pull/1215#discussion_r3855563179
##########
docs/functions-cli.md:
##########
@@ -133,6 +134,33 @@ The following table outlines the nested fields and related arguments under the `
| consumerCryptoFailureAction | ConsumerCryptoFailureAction | N/A | N/A |
+###### BatchingConfig
+
+The following table outlines the nested fields under the `batchingConfig` field of `producerConfig`.
+These settings were introduced by [PIP-401](https://github.com/apache/pulsar/blob/master/pip/pip-401.md).
+
+:::note
+
+`batchingConfig` is currently applied by **Java** functions only. The Python and Go runtimes hard-code
+batching to enabled with a 10 ms maximum publish delay and ignore these settings
+([#26390](https://github.com/apache/pulsar/issues/26390),
+[#26391](https://github.com/apache/pulsar/issues/26391)).
+
+:::
+
+| Field Name | Type | Related Command Argument | Description |
+|---------------------------|---------|--------------------------|---------------|
+| enabled | Boolean | N/A | Whether the producer batches messages. The default value is `true`. |
+| batchingMaxPublishDelayMs | Int | N/A | The maximum time the producer waits before sending a batch, in milliseconds. The default value is `10`. |
+| batchingMaxMessages | Int | N/A | The maximum number of messages in a batch. When unset, the client default applies. |
+| batchingMaxBytes | Int | N/A | The maximum size of a batch, in bytes. When unset, the client default applies. |
+| batchBuilder | String | N/A | The type of batch construction method. Available values: `DEFAULT` and `KEY_BASED`. Takes precedence over `producerConfig.batchBuilder`. |
+| roundRobinRouterBatchingPartitionSwitchFrequency | Int | N/A | How often the round-robin router switches partitions for non-keyed messages, as a multiple of `batchingMaxPublishDelayMs`. The partition switch period is `frequency * batchingMaxPublishDelayMs`. |
+
+A function that sets no `producerConfig`, or a `producerConfig` with no `batchingConfig`, gets batching
+enabled with a maximum publish delay of 10 ms. This is the behaviour functions had before these settings
+became configurable, and it is the same across the Java, Python, and Go runtimes.
Review Comment:
Spelling: the docs use American English (“behavior”) elsewhere; this new sentence introduces the British spelling “behaviour”. To stay consistent, please change it to “behavior”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][doc] Document producerConfig.batchingConfig for Pulsar Functions [pulsar-site]
Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio merged PR #1215:
URL: https://github.com/apache/pulsar-site/pull/1215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org