You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "m-ghazanfar (via GitHub)" <gi...@apache.org> on 2023/10/13 06:29:52 UTC
[I] Support secondary partitioning when ingesting from Kafka (druid)
m-ghazanfar opened a new issue, #15149:
URL: https://github.com/apache/druid/issues/15149
### Description
We use druid to store and query telemetry data. We ingest data into Druid via Kafka. Our ingestion rate is around 1.7M messages per sec.
Our query load is about `1000` qps and most queries request for for data that's in the time range [now-5, now-8].
Since we have a high ingestion rate, we produce a lot of segments. And since our queries are for real-time data, Druid ends up querying all segments within that time chunk.
If you see the below graphs, you can see that `726` queries on the broker translate to about `66.25k` queries on the indexers. Which is a fanout of about `92`. We have `94` indexer nodes.
<img width="1651" alt="Screenshot 2023-10-13 at 10 54 56 AM" src="https://github.com/apache/druid/assets/88474681/844573ff-01cb-4f33-b7c0-a8f02a86d03a">
<img width="1657" alt="Screenshot 2023-10-13 at 11 56 17 AM" src="https://github.com/apache/druid/assets/88474681/9590b56a-96bf-49ed-93cc-3786e1d09f2c">
Our data has a `tenant` dimension. The `tenant` dimension is always used to filter when performing a query.
We want to perform secondary partitioning based on the `tenant` dimension - so that the broker can prune the segments which have to be queried.
Data of one `tenant` is limited to a few kafka partitions(about 20). So, after having secondary partitioning, I would expect my fanout to be about 20, as opposed to the 92 that I am seeing now.
I know that this can be done via compaction - however, I can no make use of compaction because our queries are realtime
### Implementation
I do not have an implementation in mind but do wish to contribute the implementation myself.
### Related
- https://github.com/apache/druid/issues/12929 : not the same as this because I don't want to add the kafka partition info
- https://imply.io/blog/multi-dimensional-range-partitioning/
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15149: Support secondary partitioning when ingesting from Kafka
URL: https://github.com/apache/druid/issues/15149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-2375491829
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "etipton-splunk (via GitHub)" <gi...@apache.org>.
etipton-splunk commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-3212923290
IMO the wording should be updated [here](https://druid.apache.org/docs/latest/ingestion/partitioning/#secondary-partitioning) to clarify that "secondary partitioning" isn't truly possible at all for Kafka or Kinesis _unless_ something else occurs sometime after initial ingestion (compaction). As currently worded, it seems as if all you need to do is partition things properly within your stream, but that won't really help with query performance if I understand correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-1772334414
Ah, makes sense. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-1768962181
Thanks @abhishekagarwal87 !
I will follow up and see what's required to merge #11307
While I agree that #11307 will give me a lot of improvement, can you also help me understand why you think this is a better solution than pruning at the broker's side? Why even make the additional calls to the indexer nodes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-1770256665
The pruning on the broker's side will require the real-time segment partitions to be generated with a particular shard spec. That is tricky to implement and would only work under strict conditions (Kafka topic has to be partitioned with the same key, and re-partitioning can lead to incorrect results).
A HTTP call from broker to peon/indexer is not expensive but the query execution on the peons is not fast enough in the absence of indexes. Having a bitmap index is much more flexible from the operations point of view. The input topic can be re-partitioned without any issues, and having a bitmap would be useful in far more situations.
- It would work with any streaming connector
- It would help optimize the queries even further if you have a filter on a dimension that input data is not partitioned on.
- It doesn't necessarily require the input data to be partitioned in a specific way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-1761140827
I know I am the one who suggested something like this in the druid meetup. But thinking a bit more, what you probably need is https://github.com/apache/druid/pull/11307. With the bitmap indexes in real-time, there is no need for broker pruning, as almost all the indexers would be able to quickly return the results if they don't have the data for the tenant being queried and there is a bitmap index on the tenant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Support secondary partitioning when ingesting from Kafka (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15149:
URL: https://github.com/apache/druid/issues/15149#issuecomment-2316471818
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org