You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by saravanan kannappan <sk...@gmail.com> on 2018/04/27 02:17:20 UTC
Kafka mirror maker help
Hello,
I'm setting up mirror maker for kafka however it doesn't help
can you guide me some steps or else share the model shell script to execute
the mirror maker ?
highly appreciated your help.
Thanks
Sara
Re: Kafka mirror maker help
Posted by Peter Bukowinski <pm...@gmail.com>.
I run instances of Mirror Maker as supervisord tasks (http://supervisord.org <http://supervisord.org/>). I’d recommend looking into it. In addition to letting you sidestep the service issue, supervisord watches the processes and can auto-restart them if they stop for any reason.
—
Peter Bukowinski
> On Apr 27, 2018, at 11:58 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
> Sorry I hit send a bit too soon. I was so focused on the systemd part of
> the email and not the Mirror Maker part.
> Confluent packages include Mirror Maker but the systemd scripts are setup
> to use Confluent Replicator rather than Mirror Maker.
> My apologies.
>
> -hans
>
> /**
> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
> * hans@confluent.io (650)924-2670
> */
>
> On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
>> The latest Confluent packages now ship with systemd scripts. That is since
>> Confluent Version 4.1 - which included Apache Kafka 1.1
>>
>> -hans
>>
>> /**
>> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
>> * hans@confluent.io (650)924-2670
>> */
>>
>> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
>>
>>> Hiya,
>>>
>>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>>> packaging. I’ll reply here.
>>>
>>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>>> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
>>> but use systemd instead. Our systemd service unit (ERB template format)
>>> is
>>> here:
>>>
>>> https://github.com/wikimedia/puppet/blob/production/modules/
>>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>>
>>>
>>>
>>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
>>> wrote:
>>>
>>>> You should share related info, such source-destination Kafka versions,
>>>> sample Config or error if any.
>>>>
>>>> FYI, Go through
>>>> https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>>>>
>>>
>>
>>
Re: Kafka mirror maker help
Posted by Peter Bukowinski <pm...@gmail.com>.
I run instances of Mirror Maker as supervisord tasks (http://supervisord.org <http://supervisord.org/>). I’d recommend looking into it. In addition to letting you sidestep the service issue, supervisord watches the processes and can auto-restart them if they stop for any reason.
—
Peter Bukowinski
> On Apr 27, 2018, at 11:58 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
> Sorry I hit send a bit too soon. I was so focused on the systemd part of
> the email and not the Mirror Maker part.
> Confluent packages include Mirror Maker but the systemd scripts are setup
> to use Confluent Replicator rather than Mirror Maker.
> My apologies.
>
> -hans
>
> /**
> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
> * hans@confluent.io (650)924-2670
> */
>
> On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
>> The latest Confluent packages now ship with systemd scripts. That is since
>> Confluent Version 4.1 - which included Apache Kafka 1.1
>>
>> -hans
>>
>> /**
>> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
>> * hans@confluent.io (650)924-2670
>> */
>>
>> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
>>
>>> Hiya,
>>>
>>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>>> packaging. I’ll reply here.
>>>
>>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>>> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
>>> but use systemd instead. Our systemd service unit (ERB template format)
>>> is
>>> here:
>>>
>>> https://github.com/wikimedia/puppet/blob/production/modules/
>>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>>
>>>
>>>
>>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
>>> wrote:
>>>
>>>> You should share related info, such source-destination Kafka versions,
>>>> sample Config or error if any.
>>>>
>>>> FYI, Go through
>>>> https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>>>>
>>>
>>
>>
Re: Kafka mirror maker help
Posted by Hans Jespersen <ha...@confluent.io>.
Sorry I hit send a bit too soon. I was so focused on the systemd part of
the email and not the Mirror Maker part.
Confluent packages include Mirror Maker but the systemd scripts are setup
to use Confluent Replicator rather than Mirror Maker.
My apologies.
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen <ha...@confluent.io> wrote:
> The latest Confluent packages now ship with systemd scripts. That is since
> Confluent Version 4.1 - which included Apache Kafka 1.1
>
> -hans
>
> /**
> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
> * hans@confluent.io (650)924-2670
> */
>
> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
>
>> Hiya,
>>
>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>> packaging. I’ll reply here.
>>
>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
>> but use systemd instead. Our systemd service unit (ERB template format)
>> is
>> here:
>>
>> https://github.com/wikimedia/puppet/blob/production/modules/
>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>
>>
>>
>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
>> wrote:
>>
>> > You should share related info, such source-destination Kafka versions,
>> > sample Config or error if any.
>> >
>> > FYI, Go through
>> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>> >
>>
>
>
Re: Kafka mirror maker help
Posted by Hans Jespersen <ha...@confluent.io>.
Sorry I hit send a bit too soon. I was so focused on the systemd part of
the email and not the Mirror Maker part.
Confluent packages include Mirror Maker but the systemd scripts are setup
to use Confluent Replicator rather than Mirror Maker.
My apologies.
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen <ha...@confluent.io> wrote:
> The latest Confluent packages now ship with systemd scripts. That is since
> Confluent Version 4.1 - which included Apache Kafka 1.1
>
> -hans
>
> /**
> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
> * hans@confluent.io (650)924-2670
> */
>
> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
>
>> Hiya,
>>
>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>> packaging. I’ll reply here.
>>
>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
>> but use systemd instead. Our systemd service unit (ERB template format)
>> is
>> here:
>>
>> https://github.com/wikimedia/puppet/blob/production/modules/
>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>
>>
>>
>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
>> wrote:
>>
>> > You should share related info, such source-destination Kafka versions,
>> > sample Config or error if any.
>> >
>> > FYI, Go through
>> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>> >
>>
>
>
Re: Kafka mirror maker help
Posted by Hans Jespersen <ha...@confluent.io>.
The latest Confluent packages now ship with systemd scripts. That is since
Confluent Version 4.1 - which included Apache Kafka 1.1
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
> Hiya,
>
> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
> packaging. I’ll reply here.
>
> We now use Confluent’s Kafka debian packaging which does not (or did not?)
> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
> but use systemd instead. Our systemd service unit (ERB template format) is
> here:
>
> https://github.com/wikimedia/puppet/blob/production/
> modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>
>
>
> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
> wrote:
>
> > You should share related info, such source-destination Kafka versions,
> > sample Config or error if any.
> >
> > FYI, Go through
> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
> >
>
Re: Kafka mirror maker help
Posted by Hans Jespersen <ha...@confluent.io>.
The latest Confluent packages now ship with systemd scripts. That is since
Confluent Version 4.1 - which included Apache Kafka 1.1
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto <ot...@wikimedia.org> wrote:
> Hiya,
>
> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
> packaging. I’ll reply here.
>
> We now use Confluent’s Kafka debian packaging which does not (or did not?)
> ship with init scripts. We don’t use Sys V init.d scripts anymore either,
> but use systemd instead. Our systemd service unit (ERB template format) is
> here:
>
> https://github.com/wikimedia/puppet/blob/production/
> modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>
>
>
> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
> wrote:
>
> > You should share related info, such source-destination Kafka versions,
> > sample Config or error if any.
> >
> > FYI, Go through
> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
> >
>
Re: Kafka mirror maker help
Posted by Andrew Otto <ot...@wikimedia.org>.
Hiya,
Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
packaging. I’ll reply here.
We now use Confluent’s Kafka debian packaging which does not (or did not?)
ship with init scripts. We don’t use Sys V init.d scripts anymore either,
but use systemd instead. Our systemd service unit (ERB template format) is
here:
https://github.com/wikimedia/puppet/blob/production/modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
wrote:
> You should share related info, such source-destination Kafka versions,
> sample Config or error if any.
>
> FYI, Go through
> https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>
Re: Kafka mirror maker help
Posted by Andrew Otto <ot...@wikimedia.org>.
Hiya,
Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
packaging. I’ll reply here.
We now use Confluent’s Kafka debian packaging which does not (or did not?)
ship with init scripts. We don’t use Sys V init.d scripts anymore either,
but use systemd instead. Our systemd service unit (ERB template format) is
here:
https://github.com/wikimedia/puppet/blob/production/modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid <ja...@gmail.com>
wrote:
> You should share related info, such source-destination Kafka versions,
> sample Config or error if any.
>
> FYI, Go through
> https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>
Re: Kafka mirror maker help
Posted by Amrit Jangid <ja...@gmail.com>.
You should share related info, such source-destination Kafka versions,
sample Config or error if any.
FYI, Go through
https://kafka.apache.org/documentation/#basic_ops_mirror_maker
Re: Kafka mirror maker help
Posted by Amrit Jangid <ja...@gmail.com>.
You should share related info, such source-destination Kafka versions,
sample Config or error if any.
FYI, Go through
https://kafka.apache.org/documentation/#basic_ops_mirror_maker