You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/05/09 15:44:14 UTC
[GitHub] [druid] a2l007 opened a new issue, #12506: Introduce journaling of realtime events in Druid
a2l007 opened a new issue, #12506:
URL: https://github.com/apache/druid/issues/12506
### Motivation
Druid users today can use Kafka/Kinesis indexing service for a variety of realtime usecases and the way pipelines are setup may also vary accordingly. Customers may have their application produce events directly to Kafka which is then consumed by Druid. For such usecases, once the data is ingested into Druid there isn't a way to recover the original unaggregated events without setting up a separate exporting system such as Kafka Connect that writes to a storage sink such as HDFS or S3.
Following are a couple of usecases where it would be useful to have the original unaggregated data available:
- Misconfigured ingestion specs on Druid: The supervisor specs may not have the complete set of dimensions/aggregators defined or they may have incorrect transformSpecs or incorrect configuration on their inputFormat. In such cases, the events may get ingested to Druid and the customers may only realize the issue later on. By this time, the kafka retention period for those events probably expired and if reindexing from existing segments doesn't help, there isn't a way to fix the issue without having access to the raw events.
- In certain usecases, the upstream events are published into Kafka but Druid might be used for non-realtime usecases. It would be sufficient to run batch indexing here and middlemanagers dont have to be configured for query performance as with real-time ingestion. Having access to the raw data provides this flexibility.
For supporting such usecases that require access to the raw events, my proposal is to add support for journaling within Druid.
### Proposed changes
The journaling system will basically: read events from Kafka -> write it to local files on the middlemanager -> push the journaled files periodically to the configured deep storage. It will be implemented as a druid extension that will have the following:
- `Journaling Supervisor`: A supervisor implementation that will read offsets from the configured Kafka topic and spawn journaling tasks with start and end offsets that the task needs to read from.
- `Journal tasks` will read and persist the events locally and the written files will be pushed to the configured store.
- The journaling tasks should tap into the offset management mechanism provided by `SeekableStreamIndexTaskRunner` wherever possible.
- Journal supervisor and associated tasks run independent of the Kafka supervisor and kafka indexing tasks.
- Configurable retention periods that dictate the lifetime of the journal files on deep storage.
### Rationale
With the journaling extension, users will have an option to store their raw events to the configured store. They wouldn't need to setup an external exporting sink to journal their data. Users will not have to increase their retention periods
### Operational impact
No impact to current behavior. The journaling supervisor will be independent from the Kafka supervisors and once configured, it would require task slots on the middlemanager to run the journaling tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 closed issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
a2l007 closed issue #12506: Introduce journaling of realtime events in Druid
URL: https://github.com/apache/druid/issues/12506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1128383598
Thanks for taking a look at this!
> There's a feature offered by Confluent to make Kafka store/reload old data in S3, but AFAIK this is not part of vanilla Kafka.
>
> Your idea does have merit, but don't you think that the job of long-term storage and retrieval of raw data is best done by Kafka?
I assume you were referring to Kafka connect here, which is indeed a popular tool for streaming data into other data systems. The main motivations to go with a core journaling approach in Druid are:
- Many of the useful Kafka connect sinks such as the S3 sink has a commercial license associated with it, which maybe a deterrent for users.
- Users may not have an option to use Kafka connect sinks or other export tools if they aren't supported by the managed Kafka services they rely on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1133570763
I agree with @abhishekagarwal87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] dbardbar commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
dbardbar commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1127961552
@a2l007 - Kafka does provide a way to re-read already ingested data, and Druid even has a pretty easy way to reset the Kafka marker so to re-read all data retained by Kafka.
It is true that retention depends on exact configuration of Kafka, and if the user didn't configure it correctly, there may only a hours/days worth of data in Kafka.
There's a feature offered by Confluent to make Kafka store/reload old data in S3, but AFAIK this is not part of vanilla Kafka.
Doesn't it sound strange to you to make Druid solve the problem of long-term storage and retrieval of raw data? Why would it do a better job at this compared to Kafka?
I don't know enough about Kinesis, so I have no idea if the a similar concept applies there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1134087731
@abhishekagarwal87 Thank you for reviewing the proposal! I understand this is a niche usecase and so one of my intentions for this proposal was to see if there are similar usecases in the wild for which this idea might be useful. But with the existing workarounds and external solutions in place, looks like it might not be worth pursuing it at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1129617650
Thank you for this proposal, Atul. I have similar sentiment as @dbardbar here. We are trying to use druid to do something it is not meant for. The proposed extension targets a very particular/specific problem and is unlikely to be useful beyond those few use cases. I am afraid that if we do build this extension, it will not get regular maintenance down the road.
Thinking out loud does a setup like the below work?
- There is an ingestion job that ingests data from Kafka topic to a `raw` datasource. This ingestion doesn't use rollup and ingests all the raw data as it is and packs it into segments.
- The `raw` datasource is not stored on historicals and only on deep storage. This can be configured via load rules.
- There is also a regular ingestion pipeline that is ingesting data with rollup etc into the `queryable` datasource.
- If the rollup spec is wrong and the `queryable` datasource needs to be fixed, you can re-index data from the `raw` datasource to the `queryable` datasource.
Also, the current license with S3 Sink only aims to deter competitors and regular users can still use it for free. I have used it myself a few years ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #12506: Introduce journaling of realtime events in Druid
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1124194059
@gianm @jihoonson Any thoughts on this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org