You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/10/20 22:14:45 UTC
[GitHub] [druid] klDen opened a new issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supevisor
klDen opened a new issue #10522:
URL: https://github.com/apache/druid/issues/10522
### Affected Version
0.20.0
### Description
Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supevisor.
- Steps to reproduce the problem
1. Create 1st Kafka spec and submit;
2. Create 2nd Kafka spec and submit with exact same spec as 1st, but w/ different Kafka brokers.
Expected results: 2 Supervisors ingesting from 2 different Kafka clusters w/ same topic name to same Datasource
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-717641331
If multiple kafka clusters are needed, there's a way that I come up with to solve your problem, that is by using Kafka Mirror or uReplicator to duplicate data from source kafka clusters into one target kafka cluster from which druid ingest data
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] klDen commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
klDen commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-720615223
Thank you for the suggestions @FrankChen021, @julienlafont-tabmo .
I think the simplest solution would be to use the union query as proposed by @julienlafont-tabmo .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-716267401
Current design requires datasource as a unique key which means there's no way to create more than two supervisors with the same data source name.
Could you describe why you want to ingest from different kafka clusters instead of one to a same data source?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] klDen commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
klDen commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-717456502
Hello @FrankChen021!
For latency reasons, I currently have multiregion Kafka clusters that receives data based on location of users.
The data is sent to the proprer cluster depending on location.
For my tests, I wanted to ingest the data from those different Kafka clusters into 1 data source in Druid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 edited a comment on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
FrankChen021 edited a comment on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-716267401
Current design requires datasource as a unique key which means there's no way to create more than two supervisors with the same data source name.
Could you describe why you want to ingest from different kafka clusters instead of producing data into one kafka cluster at the producer side ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] klDen commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supevisor
Posted by GitBox <gi...@apache.org>.
klDen commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-713178006
Perhaps a solution is to specify a unique ID in the specs to differentiate them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] klDen closed issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supervisor
Posted by GitBox <gi...@apache.org>.
klDen closed issue #10522:
URL: https://github.com/apache/druid/issues/10522
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] klDen commented on issue #10522: Ingesting to same Datasource from 2 different Kafka clusters with exact same topic name/specs overwrites previous Supevisor
Posted by GitBox <gi...@apache.org>.
klDen commented on issue #10522:
URL: https://github.com/apache/druid/issues/10522#issuecomment-713836620
Seems like specifying the same Datasource using a different topic overwrites the previous Supervisor as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org