You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/20 22:18:32 UTC
[GitHub] [pulsar] nicoloboschi opened a new pull request, #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
nicoloboschi opened a new pull request, #15244:
URL: https://github.com/apache/pulsar/pull/15244
### Motivation
In the ElasticSearch sink connector the `bulkFlushIntervalInMs` is currently disabled by default.
This option is used to control the maximum time that the messages are retained in memory before being flushed to the ES server (along with `bulkActions` and `bulkSizeInMb`) when bulk mode is enabled.
`bulkActions` and `bulkSizeInMb` are based on the traffic and if `bulkFlushIntervalInMs` is disabled, the messages can be blocked in the sink forever. This is a problem because then the function subscriptions are stuck (messages are not acked since correctly sent to elastic).
`bulkFlushIntervalInMs` is a time-based setting and it will avoid this kind of scenario by design. However this could increase the overhead with low-traffic. With high traffic there should not be visible changes.
### Modifications
Enable `bulkFlushIntervalInMs` by default with 10 seconds as value. 10 seconds is just a reasonable value, nothing special behind it
- [x] `doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi merged pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
Posted by GitBox <gi...@apache.org>.
nicoloboschi merged PR #15244:
URL: https://github.com/apache/pulsar/pull/15244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on PR #15244:
URL: https://github.com/apache/pulsar/pull/15244#issuecomment-1105844632
@nicoloboschi - I added the labels that I think we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi closed pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
Posted by GitBox <gi...@apache.org>.
nicoloboschi closed pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
URL: https://github.com/apache/pulsar/pull/15244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on PR #15244:
URL: https://github.com/apache/pulsar/pull/15244#issuecomment-1104976774
@michaeljmarshall you're right. Since it is a nuance in the ElasticSearch Sink I don't think it's worth. Do we have some labels to help the release manager to highlight this pull in the release notes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on pull request #15244: [fix][elasticsearch-sink] enable bulk flushing scheduling by default
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on PR #15244:
URL: https://github.com/apache/pulsar/pull/15244#issuecomment-1112123376
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org