You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Julien C (Jira)" <ji...@apache.org> on 2020/11/14 20:41:00 UTC
[jira] [Issue Comment Deleted] (KAFKA-10710) MirrorMaker 2 creates
all combinations of herders
[ https://issues.apache.org/jira/browse/KAFKA-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien C updated KAFKA-10710:
-----------------------------
Comment: was deleted
(was: Commit is not working as expected)
> MirrorMaker 2 creates all combinations of herders
> -------------------------------------------------
>
> Key: KAFKA-10710
> URL: https://issues.apache.org/jira/browse/KAFKA-10710
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.5.1
> Reporter: Julien C
> Priority: Major
>
> We are using MM2 distributed to synchronize topics from a "Central" broker down to multiple "Local" brokers.
> {quote}replica_CENTRAL->replica_OLS.enabled = true
> replica_CENTRAL->replica_OLS.topics = _schemas
> replica_CENTRAL->replica_OLS.replication.factor = 3
> replica_CENTRAL->replica_HBG.enabled = true
> replica_CENTRAL->replica_HBG.topics = _schemas
> replica_CENTRAL->replica_HBG.replication.factor = 3
> ...
> many more
> ...
> replica_CENTRAL->replica_VIT.enabled = true
> replica_CENTRAL->replica_VIT.topics = _schemas
> replica_CENTRAL->replica_VIT.replication.factor = 3
> replica_CENTRAL->replica_UGO.enabled = true
> replica_CENTRAL->replica_UGO.topics = _schemas
> replica_CENTRAL->replica_UGO.replication.factor = 3
> {quote}
>
> When looking into the Mirror Maker logs, we discover that a herder is created for each combination even if we specifically don't describe a link between 2 clusters
>
> Exemples:
> {quote}[2020-11-12 08:43:30,351] INFO creating herder for replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:33,697] INFO creating herder for replica_CNO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:36,020] INFO creating herder for replica_CENTRAL->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:38,508] INFO creating herder for replica_UMO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:40,438] INFO creating herder for replica_CNO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:42,898] INFO creating herder for replica_ARA->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:45,457] INFO creating herder for replica_TST->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:47,860] INFO creating herder for replica_UMO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:50,248] INFO creating herder for replica_OLS->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:52,817] INFO creating herder for replica_UGO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:54,903] INFO creating herder for replica_HBG->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:57,081] INFO creating herder for replica_OLS->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:43:59,215] INFO creating herder for replica_TST->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:01,481] INFO creating herder for replica_OLS->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:03,320] INFO creating herder for replica_CNO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:05,337] INFO creating herder for replica_VIT->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:07,494] INFO creating herder for replica_CENTRAL->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:09,595] INFO creating herder for replica_ARA->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:12,201] INFO creating herder for replica_VLD->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:14,360] INFO creating herder for replica_HBG->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:16,490] INFO creating herder for replica_OLS->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:18,680] INFO creating herder for replica_ARA->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:20,779] INFO creating herder for replica_VLD->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:22,596] INFO creating herder for replica_CENTRAL->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:24,579] INFO creating herder for replica_VIT->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:26,076] INFO creating herder for replica_TST->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:27,868] INFO creating herder for replica_OLS->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:30,047] INFO creating herder for replica_TST->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:32,295] INFO creating herder for replica_OLS->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:33,777] INFO creating herder for replica_CNO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:36,083] INFO creating herder for replica_CENTRAL->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:38,857] INFO creating herder for replica_ARA->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:40,629] INFO creating herder for replica_CNO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:43,170] INFO creating herder for replica_UGO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:45,154] INFO creating herder for replica_HBG->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:46,981] INFO creating herder for replica_UMO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:49,042] INFO creating herder for replica_UGO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:51,881] INFO creating herder for replica_HBG->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:54,406] INFO creating herder for replica_HBG->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:56,378] INFO creating herder for replica_TST->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:44:58,825] INFO creating herder for replica_ARA->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:00,394] INFO creating herder for replica_CNO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:02,060] INFO creating herder for replica_UGO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:04,513] INFO creating herder for replica_TST->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:06,071] INFO creating herder for replica_VLD->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:08,700] INFO creating herder for replica_CNO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:10,533] INFO creating herder for replica_VIT->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:12,539] INFO creating herder for replica_CNO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:14,749] INFO creating herder for replica_HBG->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:16,222] INFO creating herder for replica_UMO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:18,308] INFO creating herder for replica_VIT->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:20,212] INFO creating herder for replica_HBG->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:22,618] INFO creating herder for replica_VLD->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:24,663] INFO creating herder for replica_UGO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:26,333] INFO creating herder for replica_ARA->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:28,408] INFO creating herder for replica_CNO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:30,480] INFO creating herder for replica_HBG->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:32,810] INFO creating herder for replica_VIT->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:34,904] INFO creating herder for replica_OLS->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:36,893] INFO creating herder for replica_UMO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:38,364] INFO creating herder for replica_UMO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:40,781] INFO creating herder for replica_OLS->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:43,168] INFO creating herder for replica_VIT->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:45,215] INFO creating herder for replica_CENTRAL->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:47,305] INFO creating herder for replica_UMO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 08:45:49,331] INFO creating herder for replica_VLD->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:16,537] INFO creating herder for replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:20,193] INFO creating herder for replica_CNO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:22,410] INFO creating herder for replica_CENTRAL->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:25,009] INFO creating herder for replica_UMO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:26,949] INFO creating herder for replica_CNO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:29,604] INFO creating herder for replica_ARA->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:32,121] INFO creating herder for replica_TST->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:34,883] INFO creating herder for replica_UMO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:37,323] INFO creating herder for replica_OLS->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:39,755] INFO creating herder for replica_UGO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:42,106] INFO creating herder for replica_HBG->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:44,425] INFO creating herder for replica_OLS->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:46,484] INFO creating herder for replica_TST->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:49,388] INFO creating herder for replica_OLS->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:51,300] INFO creating herder for replica_CNO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:54,560] INFO creating herder for replica_VIT->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:00:57,861] INFO creating herder for replica_CENTRAL->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:00,673] INFO creating herder for replica_ARA->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:03,222] INFO creating herder for replica_VLD->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:05,627] INFO creating herder for replica_HBG->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:07,752] INFO creating herder for replica_OLS->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:09,947] INFO creating herder for replica_ARA->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:12,492] INFO creating herder for replica_VLD->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:14,265] INFO creating herder for replica_CENTRAL->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:16,102] INFO creating herder for replica_VIT->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:17,661] INFO creating herder for replica_TST->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:19,440] INFO creating herder for replica_OLS->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:22,479] INFO creating herder for replica_TST->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:24,830] INFO creating herder for replica_OLS->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:26,478] INFO creating herder for replica_CNO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:29,179] INFO creating herder for replica_CENTRAL->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:31,616] INFO creating herder for replica_ARA->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:33,467] INFO creating herder for replica_CNO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:35,816] INFO creating herder for replica_UGO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:37,937] INFO creating herder for replica_HBG->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:40,195] INFO creating herder for replica_UMO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:42,318] INFO creating herder for replica_UGO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:44,688] INFO creating herder for replica_HBG->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:47,083] INFO creating herder for replica_HBG->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:49,199] INFO creating herder for replica_TST->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:51,516] INFO creating herder for replica_ARA->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:53,042] INFO creating herder for replica_CNO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:54,575] INFO creating herder for replica_UGO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:57,081] INFO creating herder for replica_TST->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:01:58,540] INFO creating herder for replica_VLD->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:01,584] INFO creating herder for replica_CNO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:03,401] INFO creating herder for replica_VIT->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:05,714] INFO creating herder for replica_CNO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:07,664] INFO creating herder for replica_HBG->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:09,246] INFO creating herder for replica_UMO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:11,565] INFO creating herder for replica_VIT->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:13,451] INFO creating herder for replica_HBG->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:15,872] INFO creating herder for replica_VLD->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:18,094] INFO creating herder for replica_UGO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:19,597] INFO creating herder for replica_ARA->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:21,576] INFO creating herder for replica_CNO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:23,834] INFO creating herder for replica_HBG->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:26,410] INFO creating herder for replica_VIT->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:28,516] INFO creating herder for replica_OLS->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:30,344] INFO creating herder for replica_UMO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:31,850] INFO creating herder for replica_UMO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:34,094] INFO creating herder for replica_OLS->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:36,743] INFO creating herder for replica_VIT->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:38,778] INFO creating herder for replica_CENTRAL->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:40,785] INFO creating herder for replica_UMO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:42,816] INFO creating herder for replica_VLD->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:45,225] INFO creating herder for replica_VLD->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:46,659] INFO creating herder for replica_ARA->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:48,729] INFO creating herder for replica_TST->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:50,722] INFO creating herder for replica_TST->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:52,387] INFO creating herder for replica_UGO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:54,326] INFO creating herder for replica_CENTRAL->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:56,085] INFO creating herder for replica_ARA->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:57,914] INFO creating herder for replica_VLD->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:02:59,665] INFO creating herder for replica_ARA->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:01,717] INFO creating herder for replica_OLS->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:03,832] INFO creating herder for replica_UGO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:06,278] INFO creating herder for replica_VLD->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:08,417] INFO creating herder for replica_CENTRAL->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:10,731] INFO creating herder for replica_UGO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:12,712] INFO creating herder for replica_TST->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:15,082] INFO creating herder for replica_VIT->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:17,439] INFO creating herder for replica_VLD->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:19,800] INFO creating herder for replica_HBG->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:21,554] INFO creating herder for replica_UMO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:23,536] INFO creating herder for replica_VIT->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:26,041] INFO creating herder for replica_CENTRAL->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:28,090] INFO creating herder for replica_UGO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:30,353] INFO creating herder for replica_CENTRAL->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
> [2020-11-12 09:03:32,826] INFO creating herder for replica_UMO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
> {quote}
> So much that we reached the limit of our user property LimitNOFILE recently when trying to add a new "Local" cluster.
> I believe this behavior leads to unecessary connections and resource usage that could be easily avoided by just limiting the herder creation only to elements specifically described in the mirrormaker.properties file
> [https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L130-L136]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)