You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/08/07 03:32:04 UTC
[GitHub] [druid] zrn6640 opened a new issue #10252: Kafka indexing from multiple topics into single datasource
zrn6640 opened a new issue #10252:
URL: https://github.com/apache/druid/issues/10252
Use kafka indexing to put the data of multiple kafka topics into one druid DataSource
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zrn6640 commented on issue #10252: Kafka indexing from multiple topics into single datasource
Posted by GitBox <gi...@apache.org>.
zrn6640 commented on issue #10252:
URL: https://github.com/apache/druid/issues/10252#issuecomment-671210687
Background: We need to perform statistical analysis on multiple edge data, and then display it on a large screen (the default is that one edge corresponds to a kafka topic, using druid 0.12 version, so there is no join function), so we want to implement druid The join function of (druid splits the data of multiple kafka topics through indicators (using the extension written by itself to split inside druid), and puts part of the data into a new druid dataSource, but it will also store it in their respective druid dataSource in)
Some of the fields that need to be parsed on the edge (common in kafka topic data) have specific encoding rules, and then split them into new fields and put them in the public druid dataSource
This is actually: multiple kafka topics correspond to multiple druid datasources, and now one kafka topic to multiple druid datasources have been implemented, and multiple kafka topics to one druid datasource are now discussed.
At present, the company hopes to solve it in druid. If we use your solution, it will cause Kafka to store too much data.
If you have any good ideas, you can bring them up and discuss them together
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10252: Kafka indexing from multiple topics into single datasource
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10252:
URL: https://github.com/apache/druid/issues/10252#issuecomment-671131341
What's the purpose of this requirement? Could you give more details of your use case ?
As I know in practice, different topics in Kafka usually contains different messages with different schemas, it does not make sense to ingest data from different topics into one data source. If the schemas of messages from different topics are the same, why would not you put the messages into one topic together ?
If combining the topics into one is not easy in your case, I would create a consumer application to combine messages from these different topics into one topic from which druid could ingest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org