You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampipes.apache.org by "tenthe (via GitHub)" <gi...@apache.org> on 2026/09/25 13:59:26 UTC
[I] Improve dataset write throughput in the core
tenthe opened a new issue, #5034:
URL: https://github.com/apache/streampipes/issues/5034
### Body
Raise the throughput of writing events into datasets through the REST bulk write, the CSV import and the pipeline sink by removing avoidable per-request and per-event overhead in the Influx write path.
## Description
Writing events into a dataset is limited by three sources of overhead:
- The REST bulk write endpoint POST /api/v4/datalake/measurements/{name} creates a new InfluxDB client for every request, pings it and lists databases, and the CSV import writes through the same path in 5,000-row batches. Together with the sleep on close this capped the endpoint at about 8,000 events per second regardless of batch size and the CSV import near 4,000 rows per second.
- Points are handed to the client's asynchronous batch queue, so a request returns before its data is durable.
- For every field of every event, InfluxNameSanitizer streams over the list of about 76 InfluxQL reserved keywords with a case-insensitive comparison, and the event field is removed and re-added even when its name does not change.
### StreamPipes Committer
I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Improve dataset write throughput in the core
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe closed issue #5034: Improve dataset write throughput in the core
URL: https://github.com/apache/streampipes/issues/5034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org