[GitHub] jthomas commented on issue #208: Add configurable "batch.size" property for feed subscriber

[gi...@git.apache.org]

jthomas commented on issue #208: Add configurable "batch.size" property for feed subscriber
URL: https://github.com/apache/incubator-openwhisk-package-kafka/issues/208#issuecomment-319603206
 
 
   @bjustin-ibm Using a boolean for toggling message batching would be nicer. I still think supporting a "batch.size" or "max.records" to control the batch size when `messages.batch == true` is useful. 
   
   Being able to control the parallelism when processing large amounts of messages seems really useful. Looks like this the batch size defaults to 1MB (`fetch.message.max.bytes`)
   https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md which is fairly large.
   
   Alongside making the documentation really clear on this issue, I'm wondering if there's any way we can surface these errors to the user? Thinking out loud, either pushing messages to an "error" queue or firing some kind of user-provided error trigger. There's probably a broader issue of how feed providers surface errors to clients. 
   
   Another issue to consider is, is it more efficient to still have the provider receive messages in large batches and fire triggers to the user-specified rate or just use those parameters directly in the consumer client? 
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services