You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "PramodSSImmaneni (via GitHub)" <gi...@apache.org> on 2023/03/27 17:11:52 UTC
[GitHub] [druid] PramodSSImmaneni opened a new pull request, #13982: Parallelize storage of incremental segments
PramodSSImmaneni opened a new pull request, #13982:
URL: https://github.com/apache/druid/pull/13982
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
During ingestion, incremental segments are created in memory for the different time chunks and persisted to disk when certain thresholds are reached (max number of rows, max memory, incremental persist period etc). In the case where there are a lot of dimension and metrics (1000+) it was observed that the creation/serialization of incremental segment file format for persistence and persisting the file took a while and it was blocking ingestion of new data. This affected the real-time ingestion. This serialization and persistence can be parallelized across the different time chunks. This update aims to do that.
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
The patch adds a simple configuration parameter to the ingestion tuning configuration to specify number of persistence threads. The default value is 1 if it not specified which makes it the same as it is today.
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
Added a configuration parameter to specify number of threads to use for persisting incremental segments.
-->
<hr>
##### Key changed/added classes in this PR
* `StreamAppenderator`
* `AppenderatorImpl`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [X ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1267006198
##########
docs/development/extensions-core/kafka-supervisor-reference.md:
##########
@@ -198,6 +198,7 @@ The `tuningConfig` is optional and default parameters will be used if no `tuning
| `maxTotalRows` | Long | The number of rows to aggregate across all segments; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == unlimited) |
| `intermediatePersistPeriod` | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M) |
| `maxPendingPersists` | Integer | Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with `maxRowsInMemory` * (2 + `maxPendingPersists`). | no (default == 0, meaning one persist can be running concurrently with ingestion, and none can be queued up) |
+| numPersistThreads | Integer | The number of threads to use to create and persist incremental segments on the disk. Higher ingestion data throughput results in larger number of incremental segments, causing significant cpu time to be spent on the creation of the incremental segments on the disk. For datasources with number of columns running into hundreds or thousands, creation of the incremental segments may take up significant time, in the order of multiple seconds. Both these scenarios can cause ingestion can pause frequently or stall causing it to fall behind. With more threads the segment creation can be parallelized without blocking ingestion as long as there are sufficient cpu resources available. | no (default == 1) |
Review Comment:
```suggestion
| `numPersistThreads` | Integer | The number of threads to use to create and persist incremental segments on the disk. Higher ingestion data throughput results in a larger number of incremental segments, causing significant cpu time to be spent on the creation of the incremental segments on the disk. For datasources with number of columns running into hundreds or thousands, creation of incremental segments may take up significant time, in the order of multiple seconds. In both of these scenarios, ingestion can stall or pause frequently, causing it to fall behind. You can use additional threads to parallelize the segment creation without blocking ingestion as long as there are sufficient cpu resources available. | no (default == 1) |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 merged PR #13982:
URL: https://github.com/apache/druid/pull/13982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jasonk000 commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "jasonk000 (via GitHub)" <gi...@apache.org>.
jasonk000 commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1192919413
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -247,6 +249,18 @@ public class AppenderatorImpl implements Appenderator
} else {
log.debug("Running closed segments appenderator");
}
+
+ try {
+ String pThreads = System.getProperty("druid.exp.persist.threads");
Review Comment:
I think this property should make its way to `docs/` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1267006198
##########
docs/development/extensions-core/kafka-supervisor-reference.md:
##########
@@ -198,6 +198,7 @@ The `tuningConfig` is optional and default parameters will be used if no `tuning
| `maxTotalRows` | Long | The number of rows to aggregate across all segments; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == unlimited) |
| `intermediatePersistPeriod` | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M) |
| `maxPendingPersists` | Integer | Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with `maxRowsInMemory` * (2 + `maxPendingPersists`). | no (default == 0, meaning one persist can be running concurrently with ingestion, and none can be queued up) |
+| numPersistThreads | Integer | The number of threads to use to create and persist incremental segments on the disk. Higher ingestion data throughput results in larger number of incremental segments, causing significant cpu time to be spent on the creation of the incremental segments on the disk. For datasources with number of columns running into hundreds or thousands, creation of the incremental segments may take up significant time, in the order of multiple seconds. Both these scenarios can cause ingestion can pause frequently or stall causing it to fall behind. With more threads the segment creation can be parallelized without blocking ingestion as long as there are sufficient cpu resources available. | no (default == 1) |
Review Comment:
```suggestion
| `numPersistThreads` | Integer | The number of threads to use to create and persist incremental segments on the disk. Higher ingestion data throughput results in a larger number of incremental segments, causing significant CPU time to be spent on the creation of the incremental segments on the disk. For datasources with number of columns running into hundreds or thousands, creation of incremental segments may take up significant time, in the order of multiple seconds. In both of these scenarios, ingestion can stall or pause frequently, causing it to fall behind. You can use additional threads to parallelize the segment creation without blocking ingestion as long as there are sufficient CPU resources available. | no (default == 1) |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1195393656
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -1146,9 +1160,9 @@ private void initializeExecutors()
if (persistExecutor == null) {
// use a blocking single threaded executor to throttle the firehose when write to disk is slow
persistExecutor = MoreExecutors.listeningDecorator(
- Execs.newBlockingSingleThreaded(
+ Execs.newBlockingThreaded(
"[" + StringUtils.encodeForFormat(myId) + "]-appenderator-persist",
- maxPendingPersists
+ persistThreads, maxPendingPersists
Review Comment:
I see will get back to you on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #13982: Parallelize storage of incremental segments
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1486304064
Thank you for this patch, @PramodSSImmaneni. Do you have some numbers on the time it was taking before and after the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1535139234
@abhishekagarwal87 about 400 - 500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1546126259
@kfaraz I started a thread on Druid slack in the general channel. Is there another channel that is better suited as there haven't been any comments so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13982: Parallelize storage of incremental segments
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1532446646
**Side note:**
Ideally, when you run into an issue, it's generally a good idea to go through the following steps:
- Start a thread on [Druid slack](https://druid.apache.org/community/join-slack?v=1) so that the community can help you troubleshoot your issue. We have very active community members who are always ready to discuss about new and existing problems. :)
- You may even search the [Druid slack](https://druid.apache.org/community/join-slack?v=1) and GitHub issues to see if this is a known issue.
- After discussion, if you feel this is certainly a bug/improvement, create an issue (or a PR if you already have a solution in mind) with all the details, maybe even a link to the Slack discussion.
These steps help us understand your issue and solution faster and even helps other people in the community who might be running into similar problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1890876190
Thanks, @PramodSSImmaneni , please let me know once you have pushed the latest changes. I will approve and merge the changes after that. Please try to double check the following point as well.
> In short, we need to be sure that there are no places in the code which assume persistExecutor to be single-threaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1678279956
Merged with latest and fix conflicts. Kindly let me know what else is needed for the MR to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1925448958
@kfaraz reverted changes for Kinesis and kept for Kafka. Merged latest master into it as well. Please see. Also checked again and it doesn't look like any place is relying on it being single threaded, also critical sections like persistHydrant that saves an incremental segment file and creation of commit.json that has list of incremental segments are already protected with locking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13982: Parallelize storage of incremental segments
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1532446523
@PramodSSImmaneni , 3 to 10s seems like a reasonable time for a datasource with 4000 columns.
Did you see this time increase abruptly in any specific scenario? I agree that there is room for parallelization here, but we need to understand the exact benefits that we would get from it.
It would be nice if you could share some more details on the following points:
- Did real-time ingestion slow down at any point, i.e. was there a lag buildup?
- How long did it take to persist the segments
- Once the segments were persisted, did lag catch up?
- With parallelization, how did the lag behave?
- What was your cluster setup? number of task slots, worker node types
If we do realize that parallelization is in fact needed here, it would be an MM/Indexer runtime property rather than a tuning config that has to be passed through all index specs. Users doing an ingestion need not be exposed to this detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on pull request #13982: Parallelize storage of incremental segments
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1534181626
@PramodSSImmaneni - how many of these incremental indexes were being created in an hour, in your particular case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1195392764
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -247,6 +249,18 @@ public class AppenderatorImpl implements Appenderator
} else {
log.debug("Running closed segments appenderator");
}
+
+ try {
+ String pThreads = System.getProperty("druid.exp.persist.threads");
Review Comment:
Will add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1214897259
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -247,6 +249,18 @@ public class AppenderatorImpl implements Appenderator
} else {
log.debug("Running closed segments appenderator");
}
+
+ try {
+ String pThreads = System.getProperty("druid.exp.persist.threads");
Review Comment:
This was an unused property I had added it as an earlier trial, cleaned it up. Also added documentation for the actual configuration setting @jasonk000 @abhishekagarwal87 @gianm (https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1683280882330109)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1230014565
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -247,6 +249,18 @@ public class AppenderatorImpl implements Appenderator
} else {
log.debug("Running closed segments appenderator");
}
+
+ try {
+ String pThreads = System.getProperty("druid.exp.persist.threads");
Review Comment:
Any comment @jasonk000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jasonk000 commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "jasonk000 (via GitHub)" <gi...@apache.org>.
jasonk000 commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1230252926
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -247,6 +249,18 @@ public class AppenderatorImpl implements Appenderator
} else {
log.debug("Running closed segments appenderator");
}
+
+ try {
+ String pThreads = System.getProperty("druid.exp.persist.threads");
Review Comment:
Great 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jasonk000 commented on a diff in pull request #13982: Parallelize storage of incremental segments
Posted by "jasonk000 (via GitHub)" <gi...@apache.org>.
jasonk000 commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1192923573
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -1146,9 +1160,9 @@ private void initializeExecutors()
if (persistExecutor == null) {
// use a blocking single threaded executor to throttle the firehose when write to disk is slow
persistExecutor = MoreExecutors.listeningDecorator(
- Execs.newBlockingSingleThreaded(
+ Execs.newBlockingThreaded(
"[" + StringUtils.encodeForFormat(myId) + "]-appenderator-persist",
- maxPendingPersists
+ persistThreads, maxPendingPersists
Review Comment:
I'm wary of the `persistExecutor` being called by callers assuming it is single-threaded. For example, there's nothing I can see that prevents the `close()` from occurring while a task is still running. Nothing stands out as obviously broken though. But, I do see comments like this:
```
// use persistExecutor to make sure that all the pending persists completes before
// starting to abandon segments
persistExecutor
```
Comments suggest that - if we have multiple threads - we might start processing an abandon request before or while a persist request is in queue. What scenarios/mitigations are there here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13982: Parallelize storage of incremental segments
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1546509069
@PramodSSImmaneni , We typically use the following channels:
#troubleshooting - for assistance in debugging issues
#dev - for dev proposals and discussions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1279333629
##########
docs/development/extensions-core/kafka-supervisor-reference.md:
##########
@@ -198,6 +198,7 @@ The `tuningConfig` is optional and default parameters will be used if no `tuning
| `maxTotalRows` | Long | The number of rows to aggregate across all segments; this number is post-aggregation rows. Handoff will happen either if `maxRowsPerSegment` or `maxTotalRows` is hit or every `intermediateHandoffPeriod`, whichever happens earlier. | no (default == unlimited) |
| `intermediatePersistPeriod` | ISO8601 Period | The period that determines the rate at which intermediate persists occur. | no (default == PT10M) |
| `maxPendingPersists` | Integer | Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with `maxRowsInMemory` * (2 + `maxPendingPersists`). | no (default == 0, meaning one persist can be running concurrently with ingestion, and none can be queued up) |
+| numPersistThreads | Integer | The number of threads to use to create and persist incremental segments on the disk. Higher ingestion data throughput results in larger number of incremental segments, causing significant cpu time to be spent on the creation of the incremental segments on the disk. For datasources with number of columns running into hundreds or thousands, creation of the incremental segments may take up significant time, in the order of multiple seconds. Both these scenarios can cause ingestion can pause frequently or stall causing it to fall behind. With more threads the segment creation can be parallelized without blocking ingestion as long as there are sufficient cpu resources available. | no (default == 1) |
Review Comment:
Incorporated feedback, thanks @ektravel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1925525151
@kfaraz reverted changes for Kinesis and kept for Kafka. Merged latest master into it as well. Please see. Also checked again and it doesn't look like any place is relying on it being single threaded, also critical sections like persistHydrant that saves an incremental segment file and creation of commit.json that has list of incremental segments are already protected with locking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1888304649
@PramodSSImmaneni , thanks a lot for your patience on this. Please let me know if you are still interested in taking this forward.
I'm still a little wary of this change due to reasons that @jasonk000 mentioned. In short, we need to be sure that there are no places in the code which assume `persistExecutor` to be single-threaded.
Also, in the first iteration, I would prefer if we add this new config only for the mode of ingestion in question here (i.e. Kafka, if I am not mistaken). Please remove the changes for the other modes of ingestion from this PR. If we realize that this has value for other ingestion modes, we will add the config there when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1889896170
@kfaraz yes I am still interested in taking this forward. I will make changes to restrict it to Kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1533849615
@kfaraz The slowness in creation and saving of the incremental index files (because of the large number of columns) was causing ingestion lag to increase continuously and it was falling behind more and more. It would take about 3 to 10 seconds to create the incremental index files after data had been ingested and incremental segment was ready to be persisted (because configured thresholds were reached). This would case the ingestion to fall behind even though there were available cpus on the node. There are multiple segment intervals being ingested at same time so there are multiple index files and these were being saved in a serial fashion. From what I could see there were no dependencies between them and they could be persisted parallelly.
We have a mixture of datasources and some are small that don't need this higher degree of parallelism, initially I considered a MM property but then it would apply to all datasources and a larger datasource may be starved while a smaller one is using the extra threads. Having it on a per datasource basis allows it to be configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1544286300
I have also tested with datasource with a small number of columns, whereas earlier in ingestion we were topping at around 5k thousand rows per second because of segment time spread that was over 24 hours for hourly segment granularity, with 5 persist threads it has been able to ingest at 25k rows per second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Parallelize storage of incremental segments (druid)
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on code in PR #13982:
URL: https://github.com/apache/druid/pull/13982#discussion_r1293967186
##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java:
##########
@@ -1146,9 +1160,9 @@ private void initializeExecutors()
if (persistExecutor == null) {
// use a blocking single threaded executor to throttle the firehose when write to disk is slow
persistExecutor = MoreExecutors.listeningDecorator(
- Execs.newBlockingSingleThreaded(
+ Execs.newBlockingThreaded(
"[" + StringUtils.encodeForFormat(myId) + "]-appenderator-persist",
- maxPendingPersists
+ persistThreads, maxPendingPersists
Review Comment:
Was looking at this, wouldn't the shutdown on the executor interrupt any running threads anyway. I mean even with single thread the pending persist would not complete. Also when abandoning the segments are not going to get pushed to deep storage and nor would the kafka offsets be committed isn't it. So any pending persists in the queue wouldn't matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] PramodSSImmaneni commented on pull request #13982: Parallelize storage of incremental segments
Posted by "PramodSSImmaneni (via GitHub)" <gi...@apache.org>.
PramodSSImmaneni commented on PR #13982:
URL: https://github.com/apache/druid/pull/13982#issuecomment-1513673716
Hi @abhishekagarwal87 we were using it on a datasource with about 4000 columns, with the original single thread it would take 3 - 10 seconds to create the incremental segment files. Using 4 or 5 parallelization with this change helped us handle the incoming throughput.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org