You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "m-ghazanfar (via GitHub)" <gi...@apache.org> on 2023/10/16 06:09:38 UTC
[I] Allow to retain X amount of data on indexers at handoff (druid)
m-ghazanfar opened a new issue, #15167:
URL: https://github.com/apache/druid/issues/15167
### Description
We use druid to store and query telemetry data. We ingest data into Druid via Kafka. Our ingestion rate is around 1.7M messages per sec.
Our query load is about `1000` qps and most queries request for for data that's in the time range `[now-5, now-8]`.
Since we have a high ingestion rate, we produce a lot of segments - about `2180` segments every hour. We have a segment granularity of `10` mins.
And since our queries are for real-time data, Druid ends up querying all segments within a time chunk - about `360` segments.
We have configured our Druid instance to hand off segments every `10` mins.
With this, we see a query count pattern across indexers and historicals like this,
<img width="1651" alt="Screenshot 2023-10-16 at 10 37 01 AM" src="https://github.com/apache/druid/assets/88474681/6d19fb00-a4e9-45ec-93b6-cd4c3d1db7d9">
Right after segment handoff our real-time queries go to both indexers and historicals. This is in line with the current behaviour of segment handoff - all data on the indexers gets handed-off when a handoff happens.
I would like to have a way to retain, say 10mins(or 10M rows) of data on indexers at all the times - so that I can avoid the periodic increased load on the historicals.
Adding @kfaraz because I discussed this with him in person.
With some direction, I'd be happy to implement this myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1770384952
@abhishekagarwal87 do you have comments/opinions on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15167: Allow to retain X amount of data on indexers at handoff
URL: https://github.com/apache/druid/issues/15167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1772333461
> You can still keep historical cheaper by introducing a tier that just handles very recent data
We are already doing this and want to get rid of that tier. Our query load is pretty high, at peak we get about 1000 queries per sec on the broker, about 75% of which are for realtime data. Along with this, we also have a high ingestion rate.
Also, $0 is better than cheaper :D
> This tier could be co-hosted with your indexer processes. So any additional query load is really just moving to a different process on same machine
The query load doesn't move - it expands to historicals along with indexers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1772735979
I would still consider it a "move" than "expansion" since there is less data being queried on historicals once handoff is done.
But if you insist on hacking :), you could configure your cluster to delay the handoff intentionally. You can do this through some clever setting of load rules. The segment would continue to be served by real-time nodes till the handoff is complete. You should verify that delayed handoff doesn't affect the event processing. Again, this is still a hack :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-2307949172
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1770396747
Have you looked at `maxRowsPerSegment` and `maxTotalRows` to control the handoff instead of doing it through `intermediateHandoff`? However, I am curious why an increase in historical load is a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-2253669985
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1772221758
@abhishekagarwal87 doesn't matter if you use `maxRowsPerSegment` or `maxTotalRows` - at handoff time, all segments on the indexer get handed-off. This will again result in increased load on the historicals.
Quoting from [here](https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-operations#deployment-notes-on-kafka-partitions-and-druid-segments),
> When the task reaches one of maxRowsPerSegment, maxTotalRows, or intermediateHandoffPeriod, it hands off all the segments and creates a new new set of segments will be created for further events
---
An increase in historical load is a problem because, I have to over-provision my historicals to be able to handle the additional load during the segment hand-off. If I can prevent this load on the historicals, I can move my historicals to cheaper instances or reduce their number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Allow to retain X amount of data on indexers at handoff (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #15167:
URL: https://github.com/apache/druid/issues/15167#issuecomment-1772277169
You can still keep historical cheaper by introducing a tier that just
handles very recent data. This tier could be co-hosted with your indexer
processes. So any additional query load is really just moving to a
different process on same machine.
On Fri, 20 Oct 2023 at 12:56 PM, Mohammad Ghazanfar <
***@***.***> wrote:
> @abhishekagarwal87 <https://github.com/abhishekagarwal87> doesn't matter
> if you use maxRowsPerSegment or maxTotalRows - at handoff time, all
> segments on the indexer get handed-off. This will again result in increased
> load on the historicals.
>
> Quoting from here
> <https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-operations#deployment-notes-on-kafka-partitions-and-druid-segments>
> ,
>
> When the task reaches one of maxRowsPerSegment, maxTotalRows, or
> intermediateHandoffPeriod, it hands off all the segments and creates a new
> new set of segments will be created for further events
>
> ------------------------------
>
> An increase in historical load is a problem because, I have to
> over-provision my historicals to be able to handle the additional load
> during the segment hand-off. If I can prevent this load on the historicals,
> I can move my historicals to cheaper instances or reduce their number.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/druid/issues/15167#issuecomment-1772221758>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AALIWUKPMP6I2K6IEAPSIR3YAIRUHAVCNFSM6AAAAAA6BUFOK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSGIZDCNZVHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org