You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by nitin sharma <ku...@gmail.com> on 2015/03/25 22:01:49 UTC
Kafka server relocation
Hi Team,
in my project, we have built a new datacenter for Kafka brokers and wants
to migrate from current datacenter to new one.
Switching producers and consumers wont be a problem provided New Datacenter
has all the messages of existing Datacenter.
i have only 1 topic with 2 partition that need to be migrated... that too
it is only 1 time activity.
Kindly suggest the best way to deal with this situation.
Regards,
Nitin Kumar Sharma.
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
how about the consumer lag of mirror maker?
On Mon, Apr 13, 2015 at 1:33 PM, nitin sharma <ku...@gmail.com>
wrote:
> i just tested that too and below is the stats.. it is clear that with
> "kafka-consumer-perf-test.sh", i am able to get a high throughput. around
> 44.0213 MB/sec.
>
> Seriously some configuration needs to be tweaked in MirrorMaker
> configuration for speedy processing... Can you think of something ?
>
>
> start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
> data.consumed.in.nMsg, nMsg.sec
> 2015-04-13 20:29:29:070, 2015-04-13 20:29:34:596, 1048576, 23.1552,
> 44.0213, 16683, 31716.7300
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Mon, Apr 13, 2015 at 3:51 PM, tao xiao <xi...@gmail.com> wrote:
>
> > num.consumer.fetchers means the max number of fetcher threads that can be
> > spawned. it doesn't necessarily mean you can get as many fetcher threads
> as
> > you specify.
> >
> > To me the metrics are suggesting a very slow consumption rate only 18.21
> > bytes/minute. Here is the benchmark Linkedin does
> >
> >
> >
> http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> >
> > You should check if 18.21 bytes/minute is the max throughput you can get
> on
> > your machine with bin/kafka-consumer-perf-test.sh if this is case you
> > definitely need to tune it
> >
> > On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma <
> > kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > hi Xiao,
> > >
> > > i have finally got JMX monitoring enabled for my kafka nodes in test
> > > envrionment and here is what i observed.
> > > i was monitoring mbeans under "kafka.consumer" domain of JVM running
> > "Kafka
> > > Mirror Maker" process.
> > >
> > > =========================
> > > AllTopicsBytes ===> 18.21 bytes/minute
> > > FetchRequestRateAndTimeMs ===> 9.69 Request/min and 99th Percentile is
> > > 104.13ms.
> > > ======================
> > >
> > > Interesting thing is that I have specified "num.consumer.fetchers=200"
> in
> > > my consumer property file but i can see only 8 threads of type :
> > >
> > >
> > >
> >
> "kafka.consumer":name="KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_<<zkhost>>-1428952277321-5e044226-138-1-host_<<brokerhostname>>-port_9092-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> > >
> > >
> > > Could this be the issue?
> > >
> > > note, my JVM is set to 1GB and only 30MB is utilized most of the time.
> > >
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Wed, Apr 8, 2015 at 10:48 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to
> > check
> > > if
> > > > the consumer has problem processing messages
> > > >
> > > > On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma <
> > > kumarsharma.nitin@gmail.com>
> > > > wrote:
> > > >
> > > > > thanks, but can you please tell which metrics could highlight the
> > > factor
> > > > > causing slow data migration by MirrorMaker?
> > > > >
> > > > > Regards,
> > > > > Nitin Kumar Sharma.
> > > > >
> > > > >
> > > > > On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > You may need to look into the consumer metrics and producer
> metrics
> > > to
> > > > > > identify the root cause. metrics in kafka.consumer and
> > kafka.producer
> > > > > > categories will help you find out the problems.
> > > > > >
> > > > > > This link gives instruction how to read the metrics
> > > > > > http://kafka.apache.org/documentation.html#monitoring
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <
> > > > > kumarsharma.nitin@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > hi,
> > > > > > >
> > > > > > > sorry for late response. ... i have been able to fix the issue
> ..
> > > > > problem
> > > > > > > was in my approach. I got confused between my source and target
> > > > system
> > > > > > > while defining consumer & producer property file .. it is fixed
> > now
> > > > > > >
> > > > > > > Now new issue.. the rate at which data is migrated is very very
> > > > > slow... i
> > > > > > > mean it took 5 min to copy only 15Kb.. :( .. here are the
> > property
> > > > for
> > > > > > > producer and consumer.. there is no network latency between
> > Source
> > > > and
> > > > > > > Destination clusters as such.
> > > > > > >
> > > > > > >
> > > > > > > #### Producer ###########
> > > > > > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > > > > > > serializer.class=kafka.serializer.DefaultEncoder
> > > > > > > auto.create.topics.enable=true
> > > > > > > request.required.acks=1
> > > > > > > request.required.acks=1
> > > > > > > producer.type=async
> > > > > > > batch.num.messages=3000
> > > > > > > queue.buffering.max.ms=5000
> > > > > > > queue.buffering.max.messages=100000
> > > > > > > queue.enqueue.timeout.ms=-1
> > > > > > > socket.send.buffer.bytes=5282880
> > > > > > >
> > > > > > > ######### Consumer ###########
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > > > > > > group.id=KafkaMaker
> > > > > > > auto.create.topics.enable=true
> > > > > > > socket.receive.buffer.bytes=5243880
> > > > > > > zookeeper.connection.timeout.ms=1000000
> > > > > > > num.consumer.fetchers=20
> > > > > > > fetch.message.max.bytes=5243880
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Nitin Kumar Sharma.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <
> xiaotao183@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Can you attach your mirror maker log?
> > > > > > > >
> > > > > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > > > > > > kumarsharma.nitin@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > i tried with auto.offset.reset=smallest, but still not
> > > working..
> > > > > > > > >
> > > > > > > > > there is data in my source cluster
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Nitin Kumar Sharma.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <
> > > xiaotao183@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do you have data sending to *testtopic? *By default
> mirror
> > > > maker
> > > > > > only
> > > > > > > > > > consumes data being sent after it taps into the topic.
> you
> > > need
> > > > > to
> > > > > > > keep
> > > > > > > > > > sending data to the topic after mirror maker connection
> is
> > > > > > > established.
> > > > > > > > > If
> > > > > > > > > > you want to change the behavior you can set
> > > > > > > auto.offset.reset=smallest
> > > > > > > > so
> > > > > > > > > > that any new mirror maker coming to the topic will start
> > from
> > > > the
> > > > > > > > > smallest
> > > > > > > > > > offset
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > > > > > > kumarsharma.nitin@gmail.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > thanks Xiao
> > > > > > > > > > >
> > > > > > > > > > > I tried MirrorMaker option in my test environment but
> > > > failed. I
> > > > > > am
> > > > > > > > not
> > > > > > > > > > > able to see the log getting copied to destination
> > cluster.
> > > I
> > > > > see
> > > > > > in
> > > > > > > > the
> > > > > > > > > > log
> > > > > > > > > > > of MirrorMaker process that connection is successfully
> > > > > > established
> > > > > > > > > > between
> > > > > > > > > > > source and destination cluster but still not sure what
> i
> > > > > causing
> > > > > > > the
> > > > > > > > > > problem
> > > > > > > > > > >
> > > > > > > > > > > Env. Setup ==>
> > > > > > > > > > >
> > > > > > > > > > > I). Source Cluster (Qenv02) -- i have 2
> > > > > > > broker(Qenv02kf01,Qenv02kf02)
> > > > > > > > > and
> > > > > > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > > > > > > (Qenv05kf01,Qenv05kf02)
> > > > > > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > > > > > > >
> > > > > > > > > > > II). i have kept consumer and producer properties file
> in
> > > one
> > > > > of
> > > > > > > the
> > > > > > > > > > > source kafka broker config folder.
> > > > > > > > > > >
> > > > > > > > > > > III).i have executed following command from the same
> > kafka
> > > > > broker
> > > > > > > to
> > > > > > > > > > > start the process.. log are attached :
> > > > > > > > > > >
> > > > > > > > > > > /app/kafka/bin/kafka-run-class.sh
> kafka.tools.MirrorMaker
> > > > > > > > > > -consumer.config
> > > > > > > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > > > > > > --producer.config
> > > > > > > > > > > /app/kafka/config/producer1.properties --whitelist
> > > testtopic
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > IV). I tried Consumer offset tracker tool also, while
> > > Mirror
> > > > > > Maker
> > > > > > > > > > running
> > > > > > > > > > > . I tried by launching second session of same broker
> > where
> > > > > mirror
> > > > > > > > maker
> > > > > > > > > > is
> > > > > > > > > > > running. I got error message that "*NoNode for
> > > > > > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete
> log
> > > > > > attached.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Nitin Kumar Sharma.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <
> > > > > xiaotao183@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Both consumer-1 and consumer-2 are properties of
> source
> > > > > clusters
> > > > > > > > > mirror
> > > > > > > > > > >> maker transfers data from. Mirror maker is designed to
> > be
> > > > able
> > > > > > to
> > > > > > > > > > consume
> > > > > > > > > > >> data from N sources (N >= 1) and transfer data to one
> > > > > > destination
> > > > > > > > > > cluster.
> > > > > > > > > > >> You are free to supply as many consumer properties as
> > you
> > > > want
> > > > > > to
> > > > > > > > > > instruct
> > > > > > > > > > >> mirror maker where to consumer data from.
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > > > > > > >> kumarsharma.nitin@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > > > > > > >> >
> > > > > > > > > > >> > I have actually not understood Mirror maker clearly
> > and
> > > > > hence
> > > > > > > bit
> > > > > > > > > > >> skeptical
> > > > > > > > > > >> > if i will be able to execute it effectively.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Online i hv seen the following command to be
> execute,
> > > but
> > > > > not
> > > > > > > > > > understood
> > > > > > > > > > >> > what is consumer-1 & -2.properties here? do i need
> to
> > > > copy
> > > > > > from
> > > > > > > > my
> > > > > > > > > > >> > consumer code? also, any reason why to provide
> > consumer
> > > > > > > property?
> > > > > > > > > > >> >
> > > > > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > > --consumer.config
> > > > > > > > > > >> > consumer-1.properties --consumer.config
> > > > > consumer-2.properties
> > > > > > > > > > >> > --producer.config producer.properties --whitelist
> > > my-topic
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > Regards,
> > > > > > > > > > >> > Nitin Kumar Sharma.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > You can use the Mirror maker to move data from one
> > > data
> > > > > > center
> > > > > > > > to
> > > > > > > > > > >> other
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > once all the data has been moved you can shut down
> > the
> > > > > > source
> > > > > > > > data
> > > > > > > > > > >> center
> > > > > > > > > > >> > > by doing a controlled shutdown.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Mayuresh
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > > > > > > >> <jqin@linkedin.com.invalid
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > If you want to do a seamless migration. I think
> a
> > > > better
> > > > > > way
> > > > > > > > is
> > > > > > > > > to
> > > > > > > > > > >> > build
> > > > > > > > > > >> > > a
> > > > > > > > > > >> > > > cross datacenter Kafka cluster temporarily. So
> the
> > > > > process
> > > > > > > is:
> > > > > > > > > > >> > > > 1. Add several new Kafka brokers in your new
> > > > datacenter
> > > > > > and
> > > > > > > > add
> > > > > > > > > > >> them to
> > > > > > > > > > >> > > > the old cluster.
> > > > > > > > > > >> > > > 2. Use replica assignment tool to reassign all
> the
> > > > > > > partitions
> > > > > > > > to
> > > > > > > > > > >> > brokers
> > > > > > > > > > >> > > > in new datacenter.
> > > > > > > > > > >> > > > 3. Perform controlled shutdown on the brokers in
> > old
> > > > > > > > datacenter.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > > > > > > kumarsharma.nitin@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > >Hi Team,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >in my project, we have built a new datacenter
> for
> > > > Kafka
> > > > > > > > brokers
> > > > > > > > > > and
> > > > > > > > > > >> > > wants
> > > > > > > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >Switching producers and consumers wont be a
> > problem
> > > > > > > provided
> > > > > > > > > New
> > > > > > > > > > >> > > > >Datacenter
> > > > > > > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >i have only 1 topic with 2 partition that need
> to
> > > be
> > > > > > > > > migrated...
> > > > > > > > > > >> that
> > > > > > > > > > >> > > too
> > > > > > > > > > >> > > > >it is only 1 time activity.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >Kindly suggest the best way to deal with this
> > > > > situation.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >Regards,
> > > > > > > > > > >> > > > >Nitin Kumar Sharma.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > --
> > > > > > > > > > >> > > -Regards,
> > > > > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > > > > >> > > (862) 250-7125
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Regards,
> > > > > > > > > > >> Tao
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Tao
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
i just tested that too and below is the stats.. it is clear that with
"kafka-consumer-perf-test.sh", i am able to get a high throughput. around
44.0213 MB/sec.
Seriously some configuration needs to be tweaked in MirrorMaker
configuration for speedy processing... Can you think of something ?
start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
data.consumed.in.nMsg, nMsg.sec
2015-04-13 20:29:29:070, 2015-04-13 20:29:34:596, 1048576, 23.1552,
44.0213, 16683, 31716.7300
Regards,
Nitin Kumar Sharma.
On Mon, Apr 13, 2015 at 3:51 PM, tao xiao <xi...@gmail.com> wrote:
> num.consumer.fetchers means the max number of fetcher threads that can be
> spawned. it doesn't necessarily mean you can get as many fetcher threads as
> you specify.
>
> To me the metrics are suggesting a very slow consumption rate only 18.21
> bytes/minute. Here is the benchmark Linkedin does
>
>
> http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
>
> You should check if 18.21 bytes/minute is the max throughput you can get on
> your machine with bin/kafka-consumer-perf-test.sh if this is case you
> definitely need to tune it
>
> On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> wrote:
>
> > hi Xiao,
> >
> > i have finally got JMX monitoring enabled for my kafka nodes in test
> > envrionment and here is what i observed.
> > i was monitoring mbeans under "kafka.consumer" domain of JVM running
> "Kafka
> > Mirror Maker" process.
> >
> > =========================
> > AllTopicsBytes ===> 18.21 bytes/minute
> > FetchRequestRateAndTimeMs ===> 9.69 Request/min and 99th Percentile is
> > 104.13ms.
> > ======================
> >
> > Interesting thing is that I have specified "num.consumer.fetchers=200" in
> > my consumer property file but i can see only 8 threads of type :
> >
> >
> >
> "kafka.consumer":name="KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_<<zkhost>>-1428952277321-5e044226-138-1-host_<<brokerhostname>>-port_9092-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
> >
> >
> > Could this be the issue?
> >
> > note, my JVM is set to 1GB and only 30MB is utilized most of the time.
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Wed, Apr 8, 2015 at 10:48 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to
> check
> > if
> > > the consumer has problem processing messages
> > >
> > > On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma <
> > kumarsharma.nitin@gmail.com>
> > > wrote:
> > >
> > > > thanks, but can you please tell which metrics could highlight the
> > factor
> > > > causing slow data migration by MirrorMaker?
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > > >
> > > > On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > You may need to look into the consumer metrics and producer metrics
> > to
> > > > > identify the root cause. metrics in kafka.consumer and
> kafka.producer
> > > > > categories will help you find out the problems.
> > > > >
> > > > > This link gives instruction how to read the metrics
> > > > > http://kafka.apache.org/documentation.html#monitoring
> > > > >
> > > > >
> > > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <
> > > > kumarsharma.nitin@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > hi,
> > > > > >
> > > > > > sorry for late response. ... i have been able to fix the issue ..
> > > > problem
> > > > > > was in my approach. I got confused between my source and target
> > > system
> > > > > > while defining consumer & producer property file .. it is fixed
> now
> > > > > >
> > > > > > Now new issue.. the rate at which data is migrated is very very
> > > > slow... i
> > > > > > mean it took 5 min to copy only 15Kb.. :( .. here are the
> property
> > > for
> > > > > > producer and consumer.. there is no network latency between
> Source
> > > and
> > > > > > Destination clusters as such.
> > > > > >
> > > > > >
> > > > > > #### Producer ###########
> > > > > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > > > > > serializer.class=kafka.serializer.DefaultEncoder
> > > > > > auto.create.topics.enable=true
> > > > > > request.required.acks=1
> > > > > > request.required.acks=1
> > > > > > producer.type=async
> > > > > > batch.num.messages=3000
> > > > > > queue.buffering.max.ms=5000
> > > > > > queue.buffering.max.messages=100000
> > > > > > queue.enqueue.timeout.ms=-1
> > > > > > socket.send.buffer.bytes=5282880
> > > > > >
> > > > > > ######### Consumer ###########
> > > > > >
> > > > >
> > > >
> > >
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > > > > > group.id=KafkaMaker
> > > > > > auto.create.topics.enable=true
> > > > > > socket.receive.buffer.bytes=5243880
> > > > > > zookeeper.connection.timeout.ms=1000000
> > > > > > num.consumer.fetchers=20
> > > > > > fetch.message.max.bytes=5243880
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Nitin Kumar Sharma.
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xiaotao183@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Can you attach your mirror maker log?
> > > > > > >
> > > > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > > > > > kumarsharma.nitin@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > i tried with auto.offset.reset=smallest, but still not
> > working..
> > > > > > > >
> > > > > > > > there is data in my source cluster
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Nitin Kumar Sharma.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <
> > xiaotao183@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do you have data sending to *testtopic? *By default mirror
> > > maker
> > > > > only
> > > > > > > > > consumes data being sent after it taps into the topic. you
> > need
> > > > to
> > > > > > keep
> > > > > > > > > sending data to the topic after mirror maker connection is
> > > > > > established.
> > > > > > > > If
> > > > > > > > > you want to change the behavior you can set
> > > > > > auto.offset.reset=smallest
> > > > > > > so
> > > > > > > > > that any new mirror maker coming to the topic will start
> from
> > > the
> > > > > > > > smallest
> > > > > > > > > offset
> > > > > > > > >
> > > > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > > > > > kumarsharma.nitin@gmail.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > thanks Xiao
> > > > > > > > > >
> > > > > > > > > > I tried MirrorMaker option in my test environment but
> > > failed. I
> > > > > am
> > > > > > > not
> > > > > > > > > > able to see the log getting copied to destination
> cluster.
> > I
> > > > see
> > > > > in
> > > > > > > the
> > > > > > > > > log
> > > > > > > > > > of MirrorMaker process that connection is successfully
> > > > > established
> > > > > > > > > between
> > > > > > > > > > source and destination cluster but still not sure what i
> > > > causing
> > > > > > the
> > > > > > > > > problem
> > > > > > > > > >
> > > > > > > > > > Env. Setup ==>
> > > > > > > > > >
> > > > > > > > > > I). Source Cluster (Qenv02) -- i have 2
> > > > > > broker(Qenv02kf01,Qenv02kf02)
> > > > > > > > and
> > > > > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > > > > > (Qenv05kf01,Qenv05kf02)
> > > > > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > > > > > >
> > > > > > > > > > II). i have kept consumer and producer properties file in
> > one
> > > > of
> > > > > > the
> > > > > > > > > > source kafka broker config folder.
> > > > > > > > > >
> > > > > > > > > > III).i have executed following command from the same
> kafka
> > > > broker
> > > > > > to
> > > > > > > > > > start the process.. log are attached :
> > > > > > > > > >
> > > > > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > > > > > -consumer.config
> > > > > > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > > > > > --producer.config
> > > > > > > > > > /app/kafka/config/producer1.properties --whitelist
> > testtopic
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > IV). I tried Consumer offset tracker tool also, while
> > Mirror
> > > > > Maker
> > > > > > > > > running
> > > > > > > > > > . I tried by launching second session of same broker
> where
> > > > mirror
> > > > > > > maker
> > > > > > > > > is
> > > > > > > > > > running. I got error message that "*NoNode for
> > > > > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log
> > > > > attached.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Nitin Kumar Sharma.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <
> > > > xiaotao183@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Both consumer-1 and consumer-2 are properties of source
> > > > clusters
> > > > > > > > mirror
> > > > > > > > > >> maker transfers data from. Mirror maker is designed to
> be
> > > able
> > > > > to
> > > > > > > > > consume
> > > > > > > > > >> data from N sources (N >= 1) and transfer data to one
> > > > > destination
> > > > > > > > > cluster.
> > > > > > > > > >> You are free to supply as many consumer properties as
> you
> > > want
> > > > > to
> > > > > > > > > instruct
> > > > > > > > > >> mirror maker where to consumer data from.
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > > > > > >> kumarsharma.nitin@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > > > > > >> >
> > > > > > > > > >> > I have actually not understood Mirror maker clearly
> and
> > > > hence
> > > > > > bit
> > > > > > > > > >> skeptical
> > > > > > > > > >> > if i will be able to execute it effectively.
> > > > > > > > > >> >
> > > > > > > > > >> > Online i hv seen the following command to be execute,
> > but
> > > > not
> > > > > > > > > understood
> > > > > > > > > >> > what is consumer-1 & -2.properties here? do i need to
> > > copy
> > > > > from
> > > > > > > my
> > > > > > > > > >> > consumer code? also, any reason why to provide
> consumer
> > > > > > property?
> > > > > > > > > >> >
> > > > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > --consumer.config
> > > > > > > > > >> > consumer-1.properties --consumer.config
> > > > consumer-2.properties
> > > > > > > > > >> > --producer.config producer.properties --whitelist
> > my-topic
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Regards,
> > > > > > > > > >> > Nitin Kumar Sharma.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > You can use the Mirror maker to move data from one
> > data
> > > > > center
> > > > > > > to
> > > > > > > > > >> other
> > > > > > > > > >> > and
> > > > > > > > > >> > > once all the data has been moved you can shut down
> the
> > > > > source
> > > > > > > data
> > > > > > > > > >> center
> > > > > > > > > >> > > by doing a controlled shutdown.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Mayuresh
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > > > > > >> <jqin@linkedin.com.invalid
> > > > > > > > > >> > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > If you want to do a seamless migration. I think a
> > > better
> > > > > way
> > > > > > > is
> > > > > > > > to
> > > > > > > > > >> > build
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > cross datacenter Kafka cluster temporarily. So the
> > > > process
> > > > > > is:
> > > > > > > > > >> > > > 1. Add several new Kafka brokers in your new
> > > datacenter
> > > > > and
> > > > > > > add
> > > > > > > > > >> them to
> > > > > > > > > >> > > > the old cluster.
> > > > > > > > > >> > > > 2. Use replica assignment tool to reassign all the
> > > > > > partitions
> > > > > > > to
> > > > > > > > > >> > brokers
> > > > > > > > > >> > > > in new datacenter.
> > > > > > > > > >> > > > 3. Perform controlled shutdown on the brokers in
> old
> > > > > > > datacenter.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > > > > > kumarsharma.nitin@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > >Hi Team,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >in my project, we have built a new datacenter for
> > > Kafka
> > > > > > > brokers
> > > > > > > > > and
> > > > > > > > > >> > > wants
> > > > > > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >Switching producers and consumers wont be a
> problem
> > > > > > provided
> > > > > > > > New
> > > > > > > > > >> > > > >Datacenter
> > > > > > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >i have only 1 topic with 2 partition that need to
> > be
> > > > > > > > migrated...
> > > > > > > > > >> that
> > > > > > > > > >> > > too
> > > > > > > > > >> > > > >it is only 1 time activity.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >Kindly suggest the best way to deal with this
> > > > situation.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >Regards,
> > > > > > > > > >> > > > >Nitin Kumar Sharma.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > --
> > > > > > > > > >> > > -Regards,
> > > > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > > > >> > > (862) 250-7125
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Regards,
> > > > > > > > > >> Tao
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Tao
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
num.consumer.fetchers means the max number of fetcher threads that can be
spawned. it doesn't necessarily mean you can get as many fetcher threads as
you specify.
To me the metrics are suggesting a very slow consumption rate only 18.21
bytes/minute. Here is the benchmark Linkedin does
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
You should check if 18.21 bytes/minute is the max throughput you can get on
your machine with bin/kafka-consumer-perf-test.sh if this is case you
definitely need to tune it
On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma <ku...@gmail.com>
wrote:
> hi Xiao,
>
> i have finally got JMX monitoring enabled for my kafka nodes in test
> envrionment and here is what i observed.
> i was monitoring mbeans under "kafka.consumer" domain of JVM running "Kafka
> Mirror Maker" process.
>
> =========================
> AllTopicsBytes ===> 18.21 bytes/minute
> FetchRequestRateAndTimeMs ===> 9.69 Request/min and 99th Percentile is
> 104.13ms.
> ======================
>
> Interesting thing is that I have specified "num.consumer.fetchers=200" in
> my consumer property file but i can see only 8 threads of type :
>
>
> "kafka.consumer":name="KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_<<zkhost>>-1428952277321-5e044226-138-1-host_<<brokerhostname>>-port_9092-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
>
>
> Could this be the issue?
>
> note, my JVM is set to 1GB and only 30MB is utilized most of the time.
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Wed, Apr 8, 2015 at 10:48 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check
> if
> > the consumer has problem processing messages
> >
> > On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > thanks, but can you please tell which metrics could highlight the
> factor
> > > causing slow data migration by MirrorMaker?
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > You may need to look into the consumer metrics and producer metrics
> to
> > > > identify the root cause. metrics in kafka.consumer and kafka.producer
> > > > categories will help you find out the problems.
> > > >
> > > > This link gives instruction how to read the metrics
> > > > http://kafka.apache.org/documentation.html#monitoring
> > > >
> > > >
> > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <
> > > kumarsharma.nitin@gmail.com>
> > > > wrote:
> > > >
> > > > > hi,
> > > > >
> > > > > sorry for late response. ... i have been able to fix the issue ..
> > > problem
> > > > > was in my approach. I got confused between my source and target
> > system
> > > > > while defining consumer & producer property file .. it is fixed now
> > > > >
> > > > > Now new issue.. the rate at which data is migrated is very very
> > > slow... i
> > > > > mean it took 5 min to copy only 15Kb.. :( .. here are the property
> > for
> > > > > producer and consumer.. there is no network latency between Source
> > and
> > > > > Destination clusters as such.
> > > > >
> > > > >
> > > > > #### Producer ###########
> > > > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > > > > serializer.class=kafka.serializer.DefaultEncoder
> > > > > auto.create.topics.enable=true
> > > > > request.required.acks=1
> > > > > request.required.acks=1
> > > > > producer.type=async
> > > > > batch.num.messages=3000
> > > > > queue.buffering.max.ms=5000
> > > > > queue.buffering.max.messages=100000
> > > > > queue.enqueue.timeout.ms=-1
> > > > > socket.send.buffer.bytes=5282880
> > > > >
> > > > > ######### Consumer ###########
> > > > >
> > > >
> > >
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > > > > group.id=KafkaMaker
> > > > > auto.create.topics.enable=true
> > > > > socket.receive.buffer.bytes=5243880
> > > > > zookeeper.connection.timeout.ms=1000000
> > > > > num.consumer.fetchers=20
> > > > > fetch.message.max.bytes=5243880
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Regards,
> > > > > Nitin Kumar Sharma.
> > > > >
> > > > >
> > > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Can you attach your mirror maker log?
> > > > > >
> > > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > > > > kumarsharma.nitin@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > i tried with auto.offset.reset=smallest, but still not
> working..
> > > > > > >
> > > > > > > there is data in my source cluster
> > > > > > >
> > > > > > > Regards,
> > > > > > > Nitin Kumar Sharma.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <
> xiaotao183@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Do you have data sending to *testtopic? *By default mirror
> > maker
> > > > only
> > > > > > > > consumes data being sent after it taps into the topic. you
> need
> > > to
> > > > > keep
> > > > > > > > sending data to the topic after mirror maker connection is
> > > > > established.
> > > > > > > If
> > > > > > > > you want to change the behavior you can set
> > > > > auto.offset.reset=smallest
> > > > > > so
> > > > > > > > that any new mirror maker coming to the topic will start from
> > the
> > > > > > > smallest
> > > > > > > > offset
> > > > > > > >
> > > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > > > > kumarsharma.nitin@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > thanks Xiao
> > > > > > > > >
> > > > > > > > > I tried MirrorMaker option in my test environment but
> > failed. I
> > > > am
> > > > > > not
> > > > > > > > > able to see the log getting copied to destination cluster.
> I
> > > see
> > > > in
> > > > > > the
> > > > > > > > log
> > > > > > > > > of MirrorMaker process that connection is successfully
> > > > established
> > > > > > > > between
> > > > > > > > > source and destination cluster but still not sure what i
> > > causing
> > > > > the
> > > > > > > > problem
> > > > > > > > >
> > > > > > > > > Env. Setup ==>
> > > > > > > > >
> > > > > > > > > I). Source Cluster (Qenv02) -- i have 2
> > > > > broker(Qenv02kf01,Qenv02kf02)
> > > > > > > and
> > > > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > > > > (Qenv05kf01,Qenv05kf02)
> > > > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > > > > >
> > > > > > > > > II). i have kept consumer and producer properties file in
> one
> > > of
> > > > > the
> > > > > > > > > source kafka broker config folder.
> > > > > > > > >
> > > > > > > > > III).i have executed following command from the same kafka
> > > broker
> > > > > to
> > > > > > > > > start the process.. log are attached :
> > > > > > > > >
> > > > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > > > > -consumer.config
> > > > > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > > > > --producer.config
> > > > > > > > > /app/kafka/config/producer1.properties --whitelist
> testtopic
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > IV). I tried Consumer offset tracker tool also, while
> Mirror
> > > > Maker
> > > > > > > > running
> > > > > > > > > . I tried by launching second session of same broker where
> > > mirror
> > > > > > maker
> > > > > > > > is
> > > > > > > > > running. I got error message that "*NoNode for
> > > > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log
> > > > attached.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Nitin Kumar Sharma.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <
> > > xiaotao183@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Both consumer-1 and consumer-2 are properties of source
> > > clusters
> > > > > > > mirror
> > > > > > > > >> maker transfers data from. Mirror maker is designed to be
> > able
> > > > to
> > > > > > > > consume
> > > > > > > > >> data from N sources (N >= 1) and transfer data to one
> > > > destination
> > > > > > > > cluster.
> > > > > > > > >> You are free to supply as many consumer properties as you
> > want
> > > > to
> > > > > > > > instruct
> > > > > > > > >> mirror maker where to consumer data from.
> > > > > > > > >>
> > > > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > > > > >> kumarsharma.nitin@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > > > > >> >
> > > > > > > > >> > I have actually not understood Mirror maker clearly and
> > > hence
> > > > > bit
> > > > > > > > >> skeptical
> > > > > > > > >> > if i will be able to execute it effectively.
> > > > > > > > >> >
> > > > > > > > >> > Online i hv seen the following command to be execute,
> but
> > > not
> > > > > > > > understood
> > > > > > > > >> > what is consumer-1 & -2.properties here? do i need to
> > copy
> > > > from
> > > > > > my
> > > > > > > > >> > consumer code? also, any reason why to provide consumer
> > > > > property?
> > > > > > > > >> >
> > > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > --consumer.config
> > > > > > > > >> > consumer-1.properties --consumer.config
> > > consumer-2.properties
> > > > > > > > >> > --producer.config producer.properties --whitelist
> my-topic
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Regards,
> > > > > > > > >> > Nitin Kumar Sharma.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > > > >> > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > You can use the Mirror maker to move data from one
> data
> > > > center
> > > > > > to
> > > > > > > > >> other
> > > > > > > > >> > and
> > > > > > > > >> > > once all the data has been moved you can shut down the
> > > > source
> > > > > > data
> > > > > > > > >> center
> > > > > > > > >> > > by doing a controlled shutdown.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > >
> > > > > > > > >> > > Mayuresh
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > > > > >> <jqin@linkedin.com.invalid
> > > > > > > > >> > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > If you want to do a seamless migration. I think a
> > better
> > > > way
> > > > > > is
> > > > > > > to
> > > > > > > > >> > build
> > > > > > > > >> > > a
> > > > > > > > >> > > > cross datacenter Kafka cluster temporarily. So the
> > > process
> > > > > is:
> > > > > > > > >> > > > 1. Add several new Kafka brokers in your new
> > datacenter
> > > > and
> > > > > > add
> > > > > > > > >> them to
> > > > > > > > >> > > > the old cluster.
> > > > > > > > >> > > > 2. Use replica assignment tool to reassign all the
> > > > > partitions
> > > > > > to
> > > > > > > > >> > brokers
> > > > > > > > >> > > > in new datacenter.
> > > > > > > > >> > > > 3. Perform controlled shutdown on the brokers in old
> > > > > > datacenter.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > > > >> > > >
> > > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > > > > kumarsharma.nitin@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > >Hi Team,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >in my project, we have built a new datacenter for
> > Kafka
> > > > > > brokers
> > > > > > > > and
> > > > > > > > >> > > wants
> > > > > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >Switching producers and consumers wont be a problem
> > > > > provided
> > > > > > > New
> > > > > > > > >> > > > >Datacenter
> > > > > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >i have only 1 topic with 2 partition that need to
> be
> > > > > > > migrated...
> > > > > > > > >> that
> > > > > > > > >> > > too
> > > > > > > > >> > > > >it is only 1 time activity.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >Kindly suggest the best way to deal with this
> > > situation.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >Regards,
> > > > > > > > >> > > > >Nitin Kumar Sharma.
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > --
> > > > > > > > >> > > -Regards,
> > > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > > >> > > (862) 250-7125
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Regards,
> > > > > > > > >> Tao
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
hi Xiao,
i have finally got JMX monitoring enabled for my kafka nodes in test
envrionment and here is what i observed.
i was monitoring mbeans under "kafka.consumer" domain of JVM running "Kafka
Mirror Maker" process.
=========================
AllTopicsBytes ===> 18.21 bytes/minute
FetchRequestRateAndTimeMs ===> 9.69 Request/min and 99th Percentile is
104.13ms.
======================
Interesting thing is that I have specified "num.consumer.fetchers=200" in
my consumer property file but i can see only 8 threads of type :
"kafka.consumer":name="KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_<<zkhost>>-1428952277321-5e044226-138-1-host_<<brokerhostname>>-port_9092-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics"
Could this be the issue?
note, my JVM is set to 1GB and only 30MB is utilized most of the time.
Regards,
Nitin Kumar Sharma.
On Wed, Apr 8, 2015 at 10:48 PM, tao xiao <xi...@gmail.com> wrote:
> Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
> the consumer has problem processing messages
>
> On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma <ku...@gmail.com>
> wrote:
>
> > thanks, but can you please tell which metrics could highlight the factor
> > causing slow data migration by MirrorMaker?
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > You may need to look into the consumer metrics and producer metrics to
> > > identify the root cause. metrics in kafka.consumer and kafka.producer
> > > categories will help you find out the problems.
> > >
> > > This link gives instruction how to read the metrics
> > > http://kafka.apache.org/documentation.html#monitoring
> > >
> > >
> > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <
> > kumarsharma.nitin@gmail.com>
> > > wrote:
> > >
> > > > hi,
> > > >
> > > > sorry for late response. ... i have been able to fix the issue ..
> > problem
> > > > was in my approach. I got confused between my source and target
> system
> > > > while defining consumer & producer property file .. it is fixed now
> > > >
> > > > Now new issue.. the rate at which data is migrated is very very
> > slow... i
> > > > mean it took 5 min to copy only 15Kb.. :( .. here are the property
> for
> > > > producer and consumer.. there is no network latency between Source
> and
> > > > Destination clusters as such.
> > > >
> > > >
> > > > #### Producer ###########
> > > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > > > serializer.class=kafka.serializer.DefaultEncoder
> > > > auto.create.topics.enable=true
> > > > request.required.acks=1
> > > > request.required.acks=1
> > > > producer.type=async
> > > > batch.num.messages=3000
> > > > queue.buffering.max.ms=5000
> > > > queue.buffering.max.messages=100000
> > > > queue.enqueue.timeout.ms=-1
> > > > socket.send.buffer.bytes=5282880
> > > >
> > > > ######### Consumer ###########
> > > >
> > >
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > > > group.id=KafkaMaker
> > > > auto.create.topics.enable=true
> > > > socket.receive.buffer.bytes=5243880
> > > > zookeeper.connection.timeout.ms=1000000
> > > > num.consumer.fetchers=20
> > > > fetch.message.max.bytes=5243880
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > > >
> > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Can you attach your mirror maker log?
> > > > >
> > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > > > kumarsharma.nitin@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > i tried with auto.offset.reset=smallest, but still not working..
> > > > > >
> > > > > > there is data in my source cluster
> > > > > >
> > > > > > Regards,
> > > > > > Nitin Kumar Sharma.
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xiaotao183@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Do you have data sending to *testtopic? *By default mirror
> maker
> > > only
> > > > > > > consumes data being sent after it taps into the topic. you need
> > to
> > > > keep
> > > > > > > sending data to the topic after mirror maker connection is
> > > > established.
> > > > > > If
> > > > > > > you want to change the behavior you can set
> > > > auto.offset.reset=smallest
> > > > > so
> > > > > > > that any new mirror maker coming to the topic will start from
> the
> > > > > > smallest
> > > > > > > offset
> > > > > > >
> > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > > > kumarsharma.nitin@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > thanks Xiao
> > > > > > > >
> > > > > > > > I tried MirrorMaker option in my test environment but
> failed. I
> > > am
> > > > > not
> > > > > > > > able to see the log getting copied to destination cluster. I
> > see
> > > in
> > > > > the
> > > > > > > log
> > > > > > > > of MirrorMaker process that connection is successfully
> > > established
> > > > > > > between
> > > > > > > > source and destination cluster but still not sure what i
> > causing
> > > > the
> > > > > > > problem
> > > > > > > >
> > > > > > > > Env. Setup ==>
> > > > > > > >
> > > > > > > > I). Source Cluster (Qenv02) -- i have 2
> > > > broker(Qenv02kf01,Qenv02kf02)
> > > > > > and
> > > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > > > (Qenv05kf01,Qenv05kf02)
> > > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > > > >
> > > > > > > > II). i have kept consumer and producer properties file in one
> > of
> > > > the
> > > > > > > > source kafka broker config folder.
> > > > > > > >
> > > > > > > > III).i have executed following command from the same kafka
> > broker
> > > > to
> > > > > > > > start the process.. log are attached :
> > > > > > > >
> > > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > > > -consumer.config
> > > > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > > > --producer.config
> > > > > > > > /app/kafka/config/producer1.properties --whitelist testtopic
> > > > > > > >
> > > > > > > >
> > > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror
> > > Maker
> > > > > > > running
> > > > > > > > . I tried by launching second session of same broker where
> > mirror
> > > > > maker
> > > > > > > is
> > > > > > > > running. I got error message that "*NoNode for
> > > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log
> > > attached.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Nitin Kumar Sharma.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <
> > xiaotao183@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Both consumer-1 and consumer-2 are properties of source
> > clusters
> > > > > > mirror
> > > > > > > >> maker transfers data from. Mirror maker is designed to be
> able
> > > to
> > > > > > > consume
> > > > > > > >> data from N sources (N >= 1) and transfer data to one
> > > destination
> > > > > > > cluster.
> > > > > > > >> You are free to supply as many consumer properties as you
> want
> > > to
> > > > > > > instruct
> > > > > > > >> mirror maker where to consumer data from.
> > > > > > > >>
> > > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > > > >> kumarsharma.nitin@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > > > >> >
> > > > > > > >> > I have actually not understood Mirror maker clearly and
> > hence
> > > > bit
> > > > > > > >> skeptical
> > > > > > > >> > if i will be able to execute it effectively.
> > > > > > > >> >
> > > > > > > >> > Online i hv seen the following command to be execute, but
> > not
> > > > > > > understood
> > > > > > > >> > what is consumer-1 & -2.properties here? do i need to
> copy
> > > from
> > > > > my
> > > > > > > >> > consumer code? also, any reason why to provide consumer
> > > > property?
> > > > > > > >> >
> > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > --consumer.config
> > > > > > > >> > consumer-1.properties --consumer.config
> > consumer-2.properties
> > > > > > > >> > --producer.config producer.properties --whitelist my-topic
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Regards,
> > > > > > > >> > Nitin Kumar Sharma.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > > >> > > wrote:
> > > > > > > >> >
> > > > > > > >> > > You can use the Mirror maker to move data from one data
> > > center
> > > > > to
> > > > > > > >> other
> > > > > > > >> > and
> > > > > > > >> > > once all the data has been moved you can shut down the
> > > source
> > > > > data
> > > > > > > >> center
> > > > > > > >> > > by doing a controlled shutdown.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > >
> > > > > > > >> > > Mayuresh
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > > > >> <jqin@linkedin.com.invalid
> > > > > > > >> > >
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > If you want to do a seamless migration. I think a
> better
> > > way
> > > > > is
> > > > > > to
> > > > > > > >> > build
> > > > > > > >> > > a
> > > > > > > >> > > > cross datacenter Kafka cluster temporarily. So the
> > process
> > > > is:
> > > > > > > >> > > > 1. Add several new Kafka brokers in your new
> datacenter
> > > and
> > > > > add
> > > > > > > >> them to
> > > > > > > >> > > > the old cluster.
> > > > > > > >> > > > 2. Use replica assignment tool to reassign all the
> > > > partitions
> > > > > to
> > > > > > > >> > brokers
> > > > > > > >> > > > in new datacenter.
> > > > > > > >> > > > 3. Perform controlled shutdown on the brokers in old
> > > > > datacenter.
> > > > > > > >> > > >
> > > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > > >> > > >
> > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > > > kumarsharma.nitin@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > >Hi Team,
> > > > > > > >> > > > >
> > > > > > > >> > > > >in my project, we have built a new datacenter for
> Kafka
> > > > > brokers
> > > > > > > and
> > > > > > > >> > > wants
> > > > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > > > >> > > > >
> > > > > > > >> > > > >Switching producers and consumers wont be a problem
> > > > provided
> > > > > > New
> > > > > > > >> > > > >Datacenter
> > > > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >i have only 1 topic with 2 partition that need to be
> > > > > > migrated...
> > > > > > > >> that
> > > > > > > >> > > too
> > > > > > > >> > > > >it is only 1 time activity.
> > > > > > > >> > > > >
> > > > > > > >> > > > >Kindly suggest the best way to deal with this
> > situation.
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >Regards,
> > > > > > > >> > > > >Nitin Kumar Sharma.
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > -Regards,
> > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > >> > > (862) 250-7125
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Regards,
> > > > > > > >> Tao
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
the consumer has problem processing messages
On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma <ku...@gmail.com>
wrote:
> thanks, but can you please tell which metrics could highlight the factor
> causing slow data migration by MirrorMaker?
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com> wrote:
>
> > You may need to look into the consumer metrics and producer metrics to
> > identify the root cause. metrics in kafka.consumer and kafka.producer
> > categories will help you find out the problems.
> >
> > This link gives instruction how to read the metrics
> > http://kafka.apache.org/documentation.html#monitoring
> >
> >
> > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > hi,
> > >
> > > sorry for late response. ... i have been able to fix the issue ..
> problem
> > > was in my approach. I got confused between my source and target system
> > > while defining consumer & producer property file .. it is fixed now
> > >
> > > Now new issue.. the rate at which data is migrated is very very
> slow... i
> > > mean it took 5 min to copy only 15Kb.. :( .. here are the property for
> > > producer and consumer.. there is no network latency between Source and
> > > Destination clusters as such.
> > >
> > >
> > > #### Producer ###########
> > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > > serializer.class=kafka.serializer.DefaultEncoder
> > > auto.create.topics.enable=true
> > > request.required.acks=1
> > > request.required.acks=1
> > > producer.type=async
> > > batch.num.messages=3000
> > > queue.buffering.max.ms=5000
> > > queue.buffering.max.messages=100000
> > > queue.enqueue.timeout.ms=-1
> > > socket.send.buffer.bytes=5282880
> > >
> > > ######### Consumer ###########
> > >
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > > group.id=KafkaMaker
> > > auto.create.topics.enable=true
> > > socket.receive.buffer.bytes=5243880
> > > zookeeper.connection.timeout.ms=1000000
> > > num.consumer.fetchers=20
> > > fetch.message.max.bytes=5243880
> > >
> > >
> > >
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Can you attach your mirror maker log?
> > > >
> > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > > kumarsharma.nitin@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > i tried with auto.offset.reset=smallest, but still not working..
> > > > >
> > > > > there is data in my source cluster
> > > > >
> > > > > Regards,
> > > > > Nitin Kumar Sharma.
> > > > >
> > > > >
> > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Do you have data sending to *testtopic? *By default mirror maker
> > only
> > > > > > consumes data being sent after it taps into the topic. you need
> to
> > > keep
> > > > > > sending data to the topic after mirror maker connection is
> > > established.
> > > > > If
> > > > > > you want to change the behavior you can set
> > > auto.offset.reset=smallest
> > > > so
> > > > > > that any new mirror maker coming to the topic will start from the
> > > > > smallest
> > > > > > offset
> > > > > >
> > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > > kumarsharma.nitin@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > thanks Xiao
> > > > > > >
> > > > > > > I tried MirrorMaker option in my test environment but failed. I
> > am
> > > > not
> > > > > > > able to see the log getting copied to destination cluster. I
> see
> > in
> > > > the
> > > > > > log
> > > > > > > of MirrorMaker process that connection is successfully
> > established
> > > > > > between
> > > > > > > source and destination cluster but still not sure what i
> causing
> > > the
> > > > > > problem
> > > > > > >
> > > > > > > Env. Setup ==>
> > > > > > >
> > > > > > > I). Source Cluster (Qenv02) -- i have 2
> > > broker(Qenv02kf01,Qenv02kf02)
> > > > > and
> > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > > (Qenv05kf01,Qenv05kf02)
> > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > > >
> > > > > > > II). i have kept consumer and producer properties file in one
> of
> > > the
> > > > > > > source kafka broker config folder.
> > > > > > >
> > > > > > > III).i have executed following command from the same kafka
> broker
> > > to
> > > > > > > start the process.. log are attached :
> > > > > > >
> > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > > -consumer.config
> > > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > > --producer.config
> > > > > > > /app/kafka/config/producer1.properties --whitelist testtopic
> > > > > > >
> > > > > > >
> > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror
> > Maker
> > > > > > running
> > > > > > > . I tried by launching second session of same broker where
> mirror
> > > > maker
> > > > > > is
> > > > > > > running. I got error message that "*NoNode for
> > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log
> > attached.
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Nitin Kumar Sharma.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <
> xiaotao183@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Both consumer-1 and consumer-2 are properties of source
> clusters
> > > > > mirror
> > > > > > >> maker transfers data from. Mirror maker is designed to be able
> > to
> > > > > > consume
> > > > > > >> data from N sources (N >= 1) and transfer data to one
> > destination
> > > > > > cluster.
> > > > > > >> You are free to supply as many consumer properties as you want
> > to
> > > > > > instruct
> > > > > > >> mirror maker where to consumer data from.
> > > > > > >>
> > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > > >> kumarsharma.nitin@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > > >> >
> > > > > > >> > I have actually not understood Mirror maker clearly and
> hence
> > > bit
> > > > > > >> skeptical
> > > > > > >> > if i will be able to execute it effectively.
> > > > > > >> >
> > > > > > >> > Online i hv seen the following command to be execute, but
> not
> > > > > > understood
> > > > > > >> > what is consumer-1 & -2.properties here? do i need to copy
> > from
> > > > my
> > > > > > >> > consumer code? also, any reason why to provide consumer
> > > property?
> > > > > > >> >
> > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > --consumer.config
> > > > > > >> > consumer-1.properties --consumer.config
> consumer-2.properties
> > > > > > >> > --producer.config producer.properties --whitelist my-topic
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> > Nitin Kumar Sharma.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > > >> > gharatmayuresh15@gmail.com
> > > > > > >> > > wrote:
> > > > > > >> >
> > > > > > >> > > You can use the Mirror maker to move data from one data
> > center
> > > > to
> > > > > > >> other
> > > > > > >> > and
> > > > > > >> > > once all the data has been moved you can shut down the
> > source
> > > > data
> > > > > > >> center
> > > > > > >> > > by doing a controlled shutdown.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > >
> > > > > > >> > > Mayuresh
> > > > > > >> > >
> > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > > >> <jqin@linkedin.com.invalid
> > > > > > >> > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > If you want to do a seamless migration. I think a better
> > way
> > > > is
> > > > > to
> > > > > > >> > build
> > > > > > >> > > a
> > > > > > >> > > > cross datacenter Kafka cluster temporarily. So the
> process
> > > is:
> > > > > > >> > > > 1. Add several new Kafka brokers in your new datacenter
> > and
> > > > add
> > > > > > >> them to
> > > > > > >> > > > the old cluster.
> > > > > > >> > > > 2. Use replica assignment tool to reassign all the
> > > partitions
> > > > to
> > > > > > >> > brokers
> > > > > > >> > > > in new datacenter.
> > > > > > >> > > > 3. Perform controlled shutdown on the brokers in old
> > > > datacenter.
> > > > > > >> > > >
> > > > > > >> > > > Jiangjie (Becket) Qin
> > > > > > >> > > >
> > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > > kumarsharma.nitin@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > > >
> > > > > > >> > > > >Hi Team,
> > > > > > >> > > > >
> > > > > > >> > > > >in my project, we have built a new datacenter for Kafka
> > > > brokers
> > > > > > and
> > > > > > >> > > wants
> > > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > > >> > > > >
> > > > > > >> > > > >Switching producers and consumers wont be a problem
> > > provided
> > > > > New
> > > > > > >> > > > >Datacenter
> > > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >i have only 1 topic with 2 partition that need to be
> > > > > migrated...
> > > > > > >> that
> > > > > > >> > > too
> > > > > > >> > > > >it is only 1 time activity.
> > > > > > >> > > > >
> > > > > > >> > > > >Kindly suggest the best way to deal with this
> situation.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >Regards,
> > > > > > >> > > > >Nitin Kumar Sharma.
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > -Regards,
> > > > > > >> > > Mayuresh R. Gharat
> > > > > > >> > > (862) 250-7125
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Regards,
> > > > > > >> Tao
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
thanks, but can you please tell which metrics could highlight the factor
causing slow data migration by MirrorMaker?
Regards,
Nitin Kumar Sharma.
On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xi...@gmail.com> wrote:
> You may need to look into the consumer metrics and producer metrics to
> identify the root cause. metrics in kafka.consumer and kafka.producer
> categories will help you find out the problems.
>
> This link gives instruction how to read the metrics
> http://kafka.apache.org/documentation.html#monitoring
>
>
> On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <ku...@gmail.com>
> wrote:
>
> > hi,
> >
> > sorry for late response. ... i have been able to fix the issue .. problem
> > was in my approach. I got confused between my source and target system
> > while defining consumer & producer property file .. it is fixed now
> >
> > Now new issue.. the rate at which data is migrated is very very slow... i
> > mean it took 5 min to copy only 15Kb.. :( .. here are the property for
> > producer and consumer.. there is no network latency between Source and
> > Destination clusters as such.
> >
> >
> > #### Producer ###########
> > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> > serializer.class=kafka.serializer.DefaultEncoder
> > auto.create.topics.enable=true
> > request.required.acks=1
> > request.required.acks=1
> > producer.type=async
> > batch.num.messages=3000
> > queue.buffering.max.ms=5000
> > queue.buffering.max.messages=100000
> > queue.enqueue.timeout.ms=-1
> > socket.send.buffer.bytes=5282880
> >
> > ######### Consumer ###########
> >
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> > group.id=KafkaMaker
> > auto.create.topics.enable=true
> > socket.receive.buffer.bytes=5243880
> > zookeeper.connection.timeout.ms=1000000
> > num.consumer.fetchers=20
> > fetch.message.max.bytes=5243880
> >
> >
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Can you attach your mirror maker log?
> > >
> > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> > kumarsharma.nitin@gmail.com
> > > >
> > > wrote:
> > >
> > > > i tried with auto.offset.reset=smallest, but still not working..
> > > >
> > > > there is data in my source cluster
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > > >
> > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Do you have data sending to *testtopic? *By default mirror maker
> only
> > > > > consumes data being sent after it taps into the topic. you need to
> > keep
> > > > > sending data to the topic after mirror maker connection is
> > established.
> > > > If
> > > > > you want to change the behavior you can set
> > auto.offset.reset=smallest
> > > so
> > > > > that any new mirror maker coming to the topic will start from the
> > > > smallest
> > > > > offset
> > > > >
> > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > > kumarsharma.nitin@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > thanks Xiao
> > > > > >
> > > > > > I tried MirrorMaker option in my test environment but failed. I
> am
> > > not
> > > > > > able to see the log getting copied to destination cluster. I see
> in
> > > the
> > > > > log
> > > > > > of MirrorMaker process that connection is successfully
> established
> > > > > between
> > > > > > source and destination cluster but still not sure what i causing
> > the
> > > > > problem
> > > > > >
> > > > > > Env. Setup ==>
> > > > > >
> > > > > > I). Source Cluster (Qenv02) -- i have 2
> > broker(Qenv02kf01,Qenv02kf02)
> > > > and
> > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > > (Qenv05kf01,Qenv05kf02)
> > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > > >
> > > > > > II). i have kept consumer and producer properties file in one of
> > the
> > > > > > source kafka broker config folder.
> > > > > >
> > > > > > III).i have executed following command from the same kafka broker
> > to
> > > > > > start the process.. log are attached :
> > > > > >
> > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > > -consumer.config
> > > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > > --producer.config
> > > > > > /app/kafka/config/producer1.properties --whitelist testtopic
> > > > > >
> > > > > >
> > > > > > IV). I tried Consumer offset tracker tool also, while Mirror
> Maker
> > > > > running
> > > > > > . I tried by launching second session of same broker where mirror
> > > maker
> > > > > is
> > > > > > running. I got error message that "*NoNode for
> > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log
> attached.
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Nitin Kumar Sharma.
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao183@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > >> Both consumer-1 and consumer-2 are properties of source clusters
> > > > mirror
> > > > > >> maker transfers data from. Mirror maker is designed to be able
> to
> > > > > consume
> > > > > >> data from N sources (N >= 1) and transfer data to one
> destination
> > > > > cluster.
> > > > > >> You are free to supply as many consumer properties as you want
> to
> > > > > instruct
> > > > > >> mirror maker where to consumer data from.
> > > > > >>
> > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > > >> kumarsharma.nitin@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > > >> >
> > > > > >> > I have actually not understood Mirror maker clearly and hence
> > bit
> > > > > >> skeptical
> > > > > >> > if i will be able to execute it effectively.
> > > > > >> >
> > > > > >> > Online i hv seen the following command to be execute, but not
> > > > > understood
> > > > > >> > what is consumer-1 & -2.properties here? do i need to copy
> from
> > > my
> > > > > >> > consumer code? also, any reason why to provide consumer
> > property?
> > > > > >> >
> > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
> --consumer.config
> > > > > >> > consumer-1.properties --consumer.config consumer-2.properties
> > > > > >> > --producer.config producer.properties --whitelist my-topic
> > > > > >> >
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> > Nitin Kumar Sharma.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > > >> > gharatmayuresh15@gmail.com
> > > > > >> > > wrote:
> > > > > >> >
> > > > > >> > > You can use the Mirror maker to move data from one data
> center
> > > to
> > > > > >> other
> > > > > >> > and
> > > > > >> > > once all the data has been moved you can shut down the
> source
> > > data
> > > > > >> center
> > > > > >> > > by doing a controlled shutdown.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Mayuresh
> > > > > >> > >
> > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > > >> <jqin@linkedin.com.invalid
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > If you want to do a seamless migration. I think a better
> way
> > > is
> > > > to
> > > > > >> > build
> > > > > >> > > a
> > > > > >> > > > cross datacenter Kafka cluster temporarily. So the process
> > is:
> > > > > >> > > > 1. Add several new Kafka brokers in your new datacenter
> and
> > > add
> > > > > >> them to
> > > > > >> > > > the old cluster.
> > > > > >> > > > 2. Use replica assignment tool to reassign all the
> > partitions
> > > to
> > > > > >> > brokers
> > > > > >> > > > in new datacenter.
> > > > > >> > > > 3. Perform controlled shutdown on the brokers in old
> > > datacenter.
> > > > > >> > > >
> > > > > >> > > > Jiangjie (Becket) Qin
> > > > > >> > > >
> > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > > kumarsharma.nitin@gmail.com>
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > >Hi Team,
> > > > > >> > > > >
> > > > > >> > > > >in my project, we have built a new datacenter for Kafka
> > > brokers
> > > > > and
> > > > > >> > > wants
> > > > > >> > > > >to migrate from current datacenter to new one.
> > > > > >> > > > >
> > > > > >> > > > >Switching producers and consumers wont be a problem
> > provided
> > > > New
> > > > > >> > > > >Datacenter
> > > > > >> > > > >has all the messages of existing Datacenter.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >i have only 1 topic with 2 partition that need to be
> > > > migrated...
> > > > > >> that
> > > > > >> > > too
> > > > > >> > > > >it is only 1 time activity.
> > > > > >> > > > >
> > > > > >> > > > >Kindly suggest the best way to deal with this situation.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >Regards,
> > > > > >> > > > >Nitin Kumar Sharma.
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > -Regards,
> > > > > >> > > Mayuresh R. Gharat
> > > > > >> > > (862) 250-7125
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Regards,
> > > > > >> Tao
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
You may need to look into the consumer metrics and producer metrics to
identify the root cause. metrics in kafka.consumer and kafka.producer
categories will help you find out the problems.
This link gives instruction how to read the metrics
http://kafka.apache.org/documentation.html#monitoring
On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <ku...@gmail.com>
wrote:
> hi,
>
> sorry for late response. ... i have been able to fix the issue .. problem
> was in my approach. I got confused between my source and target system
> while defining consumer & producer property file .. it is fixed now
>
> Now new issue.. the rate at which data is migrated is very very slow... i
> mean it took 5 min to copy only 15Kb.. :( .. here are the property for
> producer and consumer.. there is no network latency between Source and
> Destination clusters as such.
>
>
> #### Producer ###########
> metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
> serializer.class=kafka.serializer.DefaultEncoder
> auto.create.topics.enable=true
> request.required.acks=1
> request.required.acks=1
> producer.type=async
> batch.num.messages=3000
> queue.buffering.max.ms=5000
> queue.buffering.max.messages=100000
> queue.enqueue.timeout.ms=-1
> socket.send.buffer.bytes=5282880
>
> ######### Consumer ###########
> zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
> group.id=KafkaMaker
> auto.create.topics.enable=true
> socket.receive.buffer.bytes=5243880
> zookeeper.connection.timeout.ms=1000000
> num.consumer.fetchers=20
> fetch.message.max.bytes=5243880
>
>
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Can you attach your mirror maker log?
> >
> > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <
> kumarsharma.nitin@gmail.com
> > >
> > wrote:
> >
> > > i tried with auto.offset.reset=smallest, but still not working..
> > >
> > > there is data in my source cluster
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Do you have data sending to *testtopic? *By default mirror maker only
> > > > consumes data being sent after it taps into the topic. you need to
> keep
> > > > sending data to the topic after mirror maker connection is
> established.
> > > If
> > > > you want to change the behavior you can set
> auto.offset.reset=smallest
> > so
> > > > that any new mirror maker coming to the topic will start from the
> > > smallest
> > > > offset
> > > >
> > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > > kumarsharma.nitin@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > thanks Xiao
> > > > >
> > > > > I tried MirrorMaker option in my test environment but failed. I am
> > not
> > > > > able to see the log getting copied to destination cluster. I see in
> > the
> > > > log
> > > > > of MirrorMaker process that connection is successfully established
> > > > between
> > > > > source and destination cluster but still not sure what i causing
> the
> > > > problem
> > > > >
> > > > > Env. Setup ==>
> > > > >
> > > > > I). Source Cluster (Qenv02) -- i have 2
> broker(Qenv02kf01,Qenv02kf02)
> > > and
> > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > > Destination Clustern (Qenv05) -- i have 2 broker
> > > (Qenv05kf01,Qenv05kf02)
> > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > > >
> > > > > II). i have kept consumer and producer properties file in one of
> the
> > > > > source kafka broker config folder.
> > > > >
> > > > > III).i have executed following command from the same kafka broker
> to
> > > > > start the process.. log are attached :
> > > > >
> > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > > -consumer.config
> > > > > /app/kafka/config/consumer1.properties --num.streams=2
> > > --producer.config
> > > > > /app/kafka/config/producer1.properties --whitelist testtopic
> > > > >
> > > > >
> > > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker
> > > > running
> > > > > . I tried by launching second session of same broker where mirror
> > maker
> > > > is
> > > > > running. I got error message that "*NoNode for
> > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Nitin Kumar Sharma.
> > > > >
> > > > >
> > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Both consumer-1 and consumer-2 are properties of source clusters
> > > mirror
> > > > >> maker transfers data from. Mirror maker is designed to be able to
> > > > consume
> > > > >> data from N sources (N >= 1) and transfer data to one destination
> > > > cluster.
> > > > >> You are free to supply as many consumer properties as you want to
> > > > instruct
> > > > >> mirror maker where to consumer data from.
> > > > >>
> > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > > >> kumarsharma.nitin@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > thanks Mayuresh and Jiangjie for your response.
> > > > >> >
> > > > >> > I have actually not understood Mirror maker clearly and hence
> bit
> > > > >> skeptical
> > > > >> > if i will be able to execute it effectively.
> > > > >> >
> > > > >> > Online i hv seen the following command to be execute, but not
> > > > understood
> > > > >> > what is consumer-1 & -2.properties here? do i need to copy from
> > my
> > > > >> > consumer code? also, any reason why to provide consumer
> property?
> > > > >> >
> > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > > > >> > consumer-1.properties --consumer.config consumer-2.properties
> > > > >> > --producer.config producer.properties --whitelist my-topic
> > > > >> >
> > > > >> >
> > > > >> > Regards,
> > > > >> > Nitin Kumar Sharma.
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > > >> > gharatmayuresh15@gmail.com
> > > > >> > > wrote:
> > > > >> >
> > > > >> > > You can use the Mirror maker to move data from one data center
> > to
> > > > >> other
> > > > >> > and
> > > > >> > > once all the data has been moved you can shut down the source
> > data
> > > > >> center
> > > > >> > > by doing a controlled shutdown.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Mayuresh
> > > > >> > >
> > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > > >> <jqin@linkedin.com.invalid
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > If you want to do a seamless migration. I think a better way
> > is
> > > to
> > > > >> > build
> > > > >> > > a
> > > > >> > > > cross datacenter Kafka cluster temporarily. So the process
> is:
> > > > >> > > > 1. Add several new Kafka brokers in your new datacenter and
> > add
> > > > >> them to
> > > > >> > > > the old cluster.
> > > > >> > > > 2. Use replica assignment tool to reassign all the
> partitions
> > to
> > > > >> > brokers
> > > > >> > > > in new datacenter.
> > > > >> > > > 3. Perform controlled shutdown on the brokers in old
> > datacenter.
> > > > >> > > >
> > > > >> > > > Jiangjie (Becket) Qin
> > > > >> > > >
> > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > > kumarsharma.nitin@gmail.com>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > >Hi Team,
> > > > >> > > > >
> > > > >> > > > >in my project, we have built a new datacenter for Kafka
> > brokers
> > > > and
> > > > >> > > wants
> > > > >> > > > >to migrate from current datacenter to new one.
> > > > >> > > > >
> > > > >> > > > >Switching producers and consumers wont be a problem
> provided
> > > New
> > > > >> > > > >Datacenter
> > > > >> > > > >has all the messages of existing Datacenter.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >i have only 1 topic with 2 partition that need to be
> > > migrated...
> > > > >> that
> > > > >> > > too
> > > > >> > > > >it is only 1 time activity.
> > > > >> > > > >
> > > > >> > > > >Kindly suggest the best way to deal with this situation.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >Regards,
> > > > >> > > > >Nitin Kumar Sharma.
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > -Regards,
> > > > >> > > Mayuresh R. Gharat
> > > > >> > > (862) 250-7125
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >> Tao
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
hi,
sorry for late response. ... i have been able to fix the issue .. problem
was in my approach. I got confused between my source and target system
while defining consumer & producer property file .. it is fixed now
Now new issue.. the rate at which data is migrated is very very slow... i
mean it took 5 min to copy only 15Kb.. :( .. here are the property for
producer and consumer.. there is no network latency between Source and
Destination clusters as such.
#### Producer ###########
metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092
serializer.class=kafka.serializer.DefaultEncoder
auto.create.topics.enable=true
request.required.acks=1
request.required.acks=1
producer.type=async
batch.num.messages=3000
queue.buffering.max.ms=5000
queue.buffering.max.messages=100000
queue.enqueue.timeout.ms=-1
socket.send.buffer.bytes=5282880
######### Consumer ###########
zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181
group.id=KafkaMaker
auto.create.topics.enable=true
socket.receive.buffer.bytes=5243880
zookeeper.connection.timeout.ms=1000000
num.consumer.fetchers=20
fetch.message.max.bytes=5243880
Regards,
Nitin Kumar Sharma.
On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xi...@gmail.com> wrote:
> Can you attach your mirror maker log?
>
> On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <kumarsharma.nitin@gmail.com
> >
> wrote:
>
> > i tried with auto.offset.reset=smallest, but still not working..
> >
> > there is data in my source cluster
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Do you have data sending to *testtopic? *By default mirror maker only
> > > consumes data being sent after it taps into the topic. you need to keep
> > > sending data to the topic after mirror maker connection is established.
> > If
> > > you want to change the behavior you can set auto.offset.reset=smallest
> so
> > > that any new mirror maker coming to the topic will start from the
> > smallest
> > > offset
> > >
> > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> > kumarsharma.nitin@gmail.com
> > > >
> > > wrote:
> > >
> > > > thanks Xiao
> > > >
> > > > I tried MirrorMaker option in my test environment but failed. I am
> not
> > > > able to see the log getting copied to destination cluster. I see in
> the
> > > log
> > > > of MirrorMaker process that connection is successfully established
> > > between
> > > > source and destination cluster but still not sure what i causing the
> > > problem
> > > >
> > > > Env. Setup ==>
> > > >
> > > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02)
> > and
> > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > > Destination Clustern (Qenv05) -- i have 2 broker
> > (Qenv05kf01,Qenv05kf02)
> > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > > >
> > > > II). i have kept consumer and producer properties file in one of the
> > > > source kafka broker config folder.
> > > >
> > > > III).i have executed following command from the same kafka broker to
> > > > start the process.. log are attached :
> > > >
> > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > > -consumer.config
> > > > /app/kafka/config/consumer1.properties --num.streams=2
> > --producer.config
> > > > /app/kafka/config/producer1.properties --whitelist testtopic
> > > >
> > > >
> > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker
> > > running
> > > > . I tried by launching second session of same broker where mirror
> maker
> > > is
> > > > running. I got error message that "*NoNode for
> > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
> > > >
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > > >
> > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > >> Both consumer-1 and consumer-2 are properties of source clusters
> > mirror
> > > >> maker transfers data from. Mirror maker is designed to be able to
> > > consume
> > > >> data from N sources (N >= 1) and transfer data to one destination
> > > cluster.
> > > >> You are free to supply as many consumer properties as you want to
> > > instruct
> > > >> mirror maker where to consumer data from.
> > > >>
> > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > > >> kumarsharma.nitin@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > thanks Mayuresh and Jiangjie for your response.
> > > >> >
> > > >> > I have actually not understood Mirror maker clearly and hence bit
> > > >> skeptical
> > > >> > if i will be able to execute it effectively.
> > > >> >
> > > >> > Online i hv seen the following command to be execute, but not
> > > understood
> > > >> > what is consumer-1 & -2.properties here? do i need to copy from
> my
> > > >> > consumer code? also, any reason why to provide consumer property?
> > > >> >
> > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > > >> > consumer-1.properties --consumer.config consumer-2.properties
> > > >> > --producer.config producer.properties --whitelist my-topic
> > > >> >
> > > >> >
> > > >> > Regards,
> > > >> > Nitin Kumar Sharma.
> > > >> >
> > > >> >
> > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > > >> > gharatmayuresh15@gmail.com
> > > >> > > wrote:
> > > >> >
> > > >> > > You can use the Mirror maker to move data from one data center
> to
> > > >> other
> > > >> > and
> > > >> > > once all the data has been moved you can shut down the source
> data
> > > >> center
> > > >> > > by doing a controlled shutdown.
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Mayuresh
> > > >> > >
> > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > > >> <jqin@linkedin.com.invalid
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > If you want to do a seamless migration. I think a better way
> is
> > to
> > > >> > build
> > > >> > > a
> > > >> > > > cross datacenter Kafka cluster temporarily. So the process is:
> > > >> > > > 1. Add several new Kafka brokers in your new datacenter and
> add
> > > >> them to
> > > >> > > > the old cluster.
> > > >> > > > 2. Use replica assignment tool to reassign all the partitions
> to
> > > >> > brokers
> > > >> > > > in new datacenter.
> > > >> > > > 3. Perform controlled shutdown on the brokers in old
> datacenter.
> > > >> > > >
> > > >> > > > Jiangjie (Becket) Qin
> > > >> > > >
> > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> > kumarsharma.nitin@gmail.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > > >Hi Team,
> > > >> > > > >
> > > >> > > > >in my project, we have built a new datacenter for Kafka
> brokers
> > > and
> > > >> > > wants
> > > >> > > > >to migrate from current datacenter to new one.
> > > >> > > > >
> > > >> > > > >Switching producers and consumers wont be a problem provided
> > New
> > > >> > > > >Datacenter
> > > >> > > > >has all the messages of existing Datacenter.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >i have only 1 topic with 2 partition that need to be
> > migrated...
> > > >> that
> > > >> > > too
> > > >> > > > >it is only 1 time activity.
> > > >> > > > >
> > > >> > > > >Kindly suggest the best way to deal with this situation.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >Regards,
> > > >> > > > >Nitin Kumar Sharma.
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > -Regards,
> > > >> > > Mayuresh R. Gharat
> > > >> > > (862) 250-7125
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Regards,
> > > >> Tao
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
Can you attach your mirror maker log?
On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <ku...@gmail.com>
wrote:
> i tried with auto.offset.reset=smallest, but still not working..
>
> there is data in my source cluster
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Do you have data sending to *testtopic? *By default mirror maker only
> > consumes data being sent after it taps into the topic. you need to keep
> > sending data to the topic after mirror maker connection is established.
> If
> > you want to change the behavior you can set auto.offset.reset=smallest so
> > that any new mirror maker coming to the topic will start from the
> smallest
> > offset
> >
> > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <
> kumarsharma.nitin@gmail.com
> > >
> > wrote:
> >
> > > thanks Xiao
> > >
> > > I tried MirrorMaker option in my test environment but failed. I am not
> > > able to see the log getting copied to destination cluster. I see in the
> > log
> > > of MirrorMaker process that connection is successfully established
> > between
> > > source and destination cluster but still not sure what i causing the
> > problem
> > >
> > > Env. Setup ==>
> > >
> > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02)
> and
> > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > > Destination Clustern (Qenv05) -- i have 2 broker
> (Qenv05kf01,Qenv05kf02)
> > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> > >
> > > II). i have kept consumer and producer properties file in one of the
> > > source kafka broker config folder.
> > >
> > > III).i have executed following command from the same kafka broker to
> > > start the process.. log are attached :
> > >
> > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > -consumer.config
> > > /app/kafka/config/consumer1.properties --num.streams=2
> --producer.config
> > > /app/kafka/config/producer1.properties --whitelist testtopic
> > >
> > >
> > > IV). I tried Consumer offset tracker tool also, while Mirror Maker
> > running
> > > . I tried by launching second session of same broker where mirror maker
> > is
> > > running. I got error message that "*NoNode for
> > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
> > >
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > >> Both consumer-1 and consumer-2 are properties of source clusters
> mirror
> > >> maker transfers data from. Mirror maker is designed to be able to
> > consume
> > >> data from N sources (N >= 1) and transfer data to one destination
> > cluster.
> > >> You are free to supply as many consumer properties as you want to
> > instruct
> > >> mirror maker where to consumer data from.
> > >>
> > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> > >> kumarsharma.nitin@gmail.com>
> > >> wrote:
> > >>
> > >> > thanks Mayuresh and Jiangjie for your response.
> > >> >
> > >> > I have actually not understood Mirror maker clearly and hence bit
> > >> skeptical
> > >> > if i will be able to execute it effectively.
> > >> >
> > >> > Online i hv seen the following command to be execute, but not
> > understood
> > >> > what is consumer-1 & -2.properties here? do i need to copy from my
> > >> > consumer code? also, any reason why to provide consumer property?
> > >> >
> > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > >> > consumer-1.properties --consumer.config consumer-2.properties
> > >> > --producer.config producer.properties --whitelist my-topic
> > >> >
> > >> >
> > >> > Regards,
> > >> > Nitin Kumar Sharma.
> > >> >
> > >> >
> > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > >> > gharatmayuresh15@gmail.com
> > >> > > wrote:
> > >> >
> > >> > > You can use the Mirror maker to move data from one data center to
> > >> other
> > >> > and
> > >> > > once all the data has been moved you can shut down the source data
> > >> center
> > >> > > by doing a controlled shutdown.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Mayuresh
> > >> > >
> > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> > >> <jqin@linkedin.com.invalid
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > If you want to do a seamless migration. I think a better way is
> to
> > >> > build
> > >> > > a
> > >> > > > cross datacenter Kafka cluster temporarily. So the process is:
> > >> > > > 1. Add several new Kafka brokers in your new datacenter and add
> > >> them to
> > >> > > > the old cluster.
> > >> > > > 2. Use replica assignment tool to reassign all the partitions to
> > >> > brokers
> > >> > > > in new datacenter.
> > >> > > > 3. Perform controlled shutdown on the brokers in old datacenter.
> > >> > > >
> > >> > > > Jiangjie (Becket) Qin
> > >> > > >
> > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <
> kumarsharma.nitin@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > > >Hi Team,
> > >> > > > >
> > >> > > > >in my project, we have built a new datacenter for Kafka brokers
> > and
> > >> > > wants
> > >> > > > >to migrate from current datacenter to new one.
> > >> > > > >
> > >> > > > >Switching producers and consumers wont be a problem provided
> New
> > >> > > > >Datacenter
> > >> > > > >has all the messages of existing Datacenter.
> > >> > > > >
> > >> > > > >
> > >> > > > >i have only 1 topic with 2 partition that need to be
> migrated...
> > >> that
> > >> > > too
> > >> > > > >it is only 1 time activity.
> > >> > > > >
> > >> > > > >Kindly suggest the best way to deal with this situation.
> > >> > > > >
> > >> > > > >
> > >> > > > >Regards,
> > >> > > > >Nitin Kumar Sharma.
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > -Regards,
> > >> > > Mayuresh R. Gharat
> > >> > > (862) 250-7125
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Tao
> > >>
> > >
> > >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
i tried with auto.offset.reset=smallest, but still not working..
there is data in my source cluster
Regards,
Nitin Kumar Sharma.
On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xi...@gmail.com> wrote:
> Do you have data sending to *testtopic? *By default mirror maker only
> consumes data being sent after it taps into the topic. you need to keep
> sending data to the topic after mirror maker connection is established. If
> you want to change the behavior you can set auto.offset.reset=smallest so
> that any new mirror maker coming to the topic will start from the smallest
> offset
>
> On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <kumarsharma.nitin@gmail.com
> >
> wrote:
>
> > thanks Xiao
> >
> > I tried MirrorMaker option in my test environment but failed. I am not
> > able to see the log getting copied to destination cluster. I see in the
> log
> > of MirrorMaker process that connection is successfully established
> between
> > source and destination cluster but still not sure what i causing the
> problem
> >
> > Env. Setup ==>
> >
> > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and
> > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> > Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02)
> > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
> >
> > II). i have kept consumer and producer properties file in one of the
> > source kafka broker config folder.
> >
> > III).i have executed following command from the same kafka broker to
> > start the process.. log are attached :
> >
> > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> -consumer.config
> > /app/kafka/config/consumer1.properties --num.streams=2 --producer.config
> > /app/kafka/config/producer1.properties --whitelist testtopic
> >
> >
> > IV). I tried Consumer offset tracker tool also, while Mirror Maker
> running
> > . I tried by launching second session of same broker where mirror maker
> is
> > running. I got error message that "*NoNode for
> > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com> wrote:
> >
> >> Both consumer-1 and consumer-2 are properties of source clusters mirror
> >> maker transfers data from. Mirror maker is designed to be able to
> consume
> >> data from N sources (N >= 1) and transfer data to one destination
> cluster.
> >> You are free to supply as many consumer properties as you want to
> instruct
> >> mirror maker where to consumer data from.
> >>
> >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
> >> kumarsharma.nitin@gmail.com>
> >> wrote:
> >>
> >> > thanks Mayuresh and Jiangjie for your response.
> >> >
> >> > I have actually not understood Mirror maker clearly and hence bit
> >> skeptical
> >> > if i will be able to execute it effectively.
> >> >
> >> > Online i hv seen the following command to be execute, but not
> understood
> >> > what is consumer-1 & -2.properties here? do i need to copy from my
> >> > consumer code? also, any reason why to provide consumer property?
> >> >
> >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> >> > consumer-1.properties --consumer.config consumer-2.properties
> >> > --producer.config producer.properties --whitelist my-topic
> >> >
> >> >
> >> > Regards,
> >> > Nitin Kumar Sharma.
> >> >
> >> >
> >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> >> > gharatmayuresh15@gmail.com
> >> > > wrote:
> >> >
> >> > > You can use the Mirror maker to move data from one data center to
> >> other
> >> > and
> >> > > once all the data has been moved you can shut down the source data
> >> center
> >> > > by doing a controlled shutdown.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Mayuresh
> >> > >
> >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> >> <jqin@linkedin.com.invalid
> >> > >
> >> > > wrote:
> >> > >
> >> > > > If you want to do a seamless migration. I think a better way is to
> >> > build
> >> > > a
> >> > > > cross datacenter Kafka cluster temporarily. So the process is:
> >> > > > 1. Add several new Kafka brokers in your new datacenter and add
> >> them to
> >> > > > the old cluster.
> >> > > > 2. Use replica assignment tool to reassign all the partitions to
> >> > brokers
> >> > > > in new datacenter.
> >> > > > 3. Perform controlled shutdown on the brokers in old datacenter.
> >> > > >
> >> > > > Jiangjie (Becket) Qin
> >> > > >
> >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > >Hi Team,
> >> > > > >
> >> > > > >in my project, we have built a new datacenter for Kafka brokers
> and
> >> > > wants
> >> > > > >to migrate from current datacenter to new one.
> >> > > > >
> >> > > > >Switching producers and consumers wont be a problem provided New
> >> > > > >Datacenter
> >> > > > >has all the messages of existing Datacenter.
> >> > > > >
> >> > > > >
> >> > > > >i have only 1 topic with 2 partition that need to be migrated...
> >> that
> >> > > too
> >> > > > >it is only 1 time activity.
> >> > > > >
> >> > > > >Kindly suggest the best way to deal with this situation.
> >> > > > >
> >> > > > >
> >> > > > >Regards,
> >> > > > >Nitin Kumar Sharma.
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > -Regards,
> >> > > Mayuresh R. Gharat
> >> > > (862) 250-7125
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Tao
> >>
> >
> >
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
Do you have data sending to *testtopic? *By default mirror maker only
consumes data being sent after it taps into the topic. you need to keep
sending data to the topic after mirror maker connection is established. If
you want to change the behavior you can set auto.offset.reset=smallest so
that any new mirror maker coming to the topic will start from the smallest
offset
On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <ku...@gmail.com>
wrote:
> thanks Xiao
>
> I tried MirrorMaker option in my test environment but failed. I am not
> able to see the log getting copied to destination cluster. I see in the log
> of MirrorMaker process that connection is successfully established between
> source and destination cluster but still not sure what i causing the problem
>
> Env. Setup ==>
>
> I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and
> 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
> Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02)
> and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
>
> II). i have kept consumer and producer properties file in one of the
> source kafka broker config folder.
>
> III).i have executed following command from the same kafka broker to
> start the process.. log are attached :
>
> /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker -consumer.config
> /app/kafka/config/consumer1.properties --num.streams=2 --producer.config
> /app/kafka/config/producer1.properties --whitelist testtopic
>
>
> IV). I tried Consumer offset tracker tool also, while Mirror Maker running
> . I tried by launching second session of same broker where mirror maker is
> running. I got error message that "*NoNode for
> /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com> wrote:
>
>> Both consumer-1 and consumer-2 are properties of source clusters mirror
>> maker transfers data from. Mirror maker is designed to be able to consume
>> data from N sources (N >= 1) and transfer data to one destination cluster.
>> You are free to supply as many consumer properties as you want to instruct
>> mirror maker where to consumer data from.
>>
>> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <
>> kumarsharma.nitin@gmail.com>
>> wrote:
>>
>> > thanks Mayuresh and Jiangjie for your response.
>> >
>> > I have actually not understood Mirror maker clearly and hence bit
>> skeptical
>> > if i will be able to execute it effectively.
>> >
>> > Online i hv seen the following command to be execute, but not understood
>> > what is consumer-1 & -2.properties here? do i need to copy from my
>> > consumer code? also, any reason why to provide consumer property?
>> >
>> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
>> > consumer-1.properties --consumer.config consumer-2.properties
>> > --producer.config producer.properties --whitelist my-topic
>> >
>> >
>> > Regards,
>> > Nitin Kumar Sharma.
>> >
>> >
>> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
>> > gharatmayuresh15@gmail.com
>> > > wrote:
>> >
>> > > You can use the Mirror maker to move data from one data center to
>> other
>> > and
>> > > once all the data has been moved you can shut down the source data
>> center
>> > > by doing a controlled shutdown.
>> > >
>> > > Thanks,
>> > >
>> > > Mayuresh
>> > >
>> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
>> <jqin@linkedin.com.invalid
>> > >
>> > > wrote:
>> > >
>> > > > If you want to do a seamless migration. I think a better way is to
>> > build
>> > > a
>> > > > cross datacenter Kafka cluster temporarily. So the process is:
>> > > > 1. Add several new Kafka brokers in your new datacenter and add
>> them to
>> > > > the old cluster.
>> > > > 2. Use replica assignment tool to reassign all the partitions to
>> > brokers
>> > > > in new datacenter.
>> > > > 3. Perform controlled shutdown on the brokers in old datacenter.
>> > > >
>> > > > Jiangjie (Becket) Qin
>> > > >
>> > > > On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com>
>> > wrote:
>> > > >
>> > > > >Hi Team,
>> > > > >
>> > > > >in my project, we have built a new datacenter for Kafka brokers and
>> > > wants
>> > > > >to migrate from current datacenter to new one.
>> > > > >
>> > > > >Switching producers and consumers wont be a problem provided New
>> > > > >Datacenter
>> > > > >has all the messages of existing Datacenter.
>> > > > >
>> > > > >
>> > > > >i have only 1 topic with 2 partition that need to be migrated...
>> that
>> > > too
>> > > > >it is only 1 time activity.
>> > > > >
>> > > > >Kindly suggest the best way to deal with this situation.
>> > > > >
>> > > > >
>> > > > >Regards,
>> > > > >Nitin Kumar Sharma.
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > -Regards,
>> > > Mayuresh R. Gharat
>> > > (862) 250-7125
>> > >
>> >
>>
>>
>>
>> --
>> Regards,
>> Tao
>>
>
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
thanks Xiao
I tried MirrorMaker option in my test environment but failed. I am not able
to see the log getting copied to destination cluster. I see in the log of
MirrorMaker process that connection is successfully established between
source and destination cluster but still not sure what i causing the problem
Env. Setup ==>
I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and 3
zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02)
and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
II). i have kept consumer and producer properties file in one of the source
kafka broker config folder.
III).i have executed following command from the same kafka broker to start
the process.. log are attached :
/app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker -consumer.config
/app/kafka/config/consumer1.properties --num.streams=2 --producer.config
/app/kafka/config/producer1.properties --whitelist testtopic
IV). I tried Consumer offset tracker tool also, while Mirror Maker running
. I tried by launching second session of same broker where mirror maker is
running. I got error message that "*NoNode for
/consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
Regards,
Nitin Kumar Sharma.
On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xi...@gmail.com> wrote:
> Both consumer-1 and consumer-2 are properties of source clusters mirror
> maker transfers data from. Mirror maker is designed to be able to consume
> data from N sources (N >= 1) and transfer data to one destination cluster.
> You are free to supply as many consumer properties as you want to instruct
> mirror maker where to consumer data from.
>
> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <kumarsharma.nitin@gmail.com
> >
> wrote:
>
> > thanks Mayuresh and Jiangjie for your response.
> >
> > I have actually not understood Mirror maker clearly and hence bit
> skeptical
> > if i will be able to execute it effectively.
> >
> > Online i hv seen the following command to be execute, but not understood
> > what is consumer-1 & -2.properties here? do i need to copy from my
> > consumer code? also, any reason why to provide consumer property?
> >
> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > consumer-1.properties --consumer.config consumer-2.properties
> > --producer.config producer.properties --whitelist my-topic
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com
> > > wrote:
> >
> > > You can use the Mirror maker to move data from one data center to other
> > and
> > > once all the data has been moved you can shut down the source data
> center
> > > by doing a controlled shutdown.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> <jqin@linkedin.com.invalid
> > >
> > > wrote:
> > >
> > > > If you want to do a seamless migration. I think a better way is to
> > build
> > > a
> > > > cross datacenter Kafka cluster temporarily. So the process is:
> > > > 1. Add several new Kafka brokers in your new datacenter and add them
> to
> > > > the old cluster.
> > > > 2. Use replica assignment tool to reassign all the partitions to
> > brokers
> > > > in new datacenter.
> > > > 3. Perform controlled shutdown on the brokers in old datacenter.
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com>
> > wrote:
> > > >
> > > > >Hi Team,
> > > > >
> > > > >in my project, we have built a new datacenter for Kafka brokers and
> > > wants
> > > > >to migrate from current datacenter to new one.
> > > > >
> > > > >Switching producers and consumers wont be a problem provided New
> > > > >Datacenter
> > > > >has all the messages of existing Datacenter.
> > > > >
> > > > >
> > > > >i have only 1 topic with 2 partition that need to be migrated...
> that
> > > too
> > > > >it is only 1 time activity.
> > > > >
> > > > >Kindly suggest the best way to deal with this situation.
> > > > >
> > > > >
> > > > >Regards,
> > > > >Nitin Kumar Sharma.
> > > >
> > > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Kafka server relocation
Posted by tao xiao <xi...@gmail.com>.
Both consumer-1 and consumer-2 are properties of source clusters mirror
maker transfers data from. Mirror maker is designed to be able to consume
data from N sources (N >= 1) and transfer data to one destination cluster.
You are free to supply as many consumer properties as you want to instruct
mirror maker where to consumer data from.
On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <ku...@gmail.com>
wrote:
> thanks Mayuresh and Jiangjie for your response.
>
> I have actually not understood Mirror maker clearly and hence bit skeptical
> if i will be able to execute it effectively.
>
> Online i hv seen the following command to be execute, but not understood
> what is consumer-1 & -2.properties here? do i need to copy from my
> consumer code? also, any reason why to provide consumer property?
>
> bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> consumer-1.properties --consumer.config consumer-2.properties
> --producer.config producer.properties --whitelist my-topic
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> > wrote:
>
> > You can use the Mirror maker to move data from one data center to other
> and
> > once all the data has been moved you can shut down the source data center
> > by doing a controlled shutdown.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > wrote:
> >
> > > If you want to do a seamless migration. I think a better way is to
> build
> > a
> > > cross datacenter Kafka cluster temporarily. So the process is:
> > > 1. Add several new Kafka brokers in your new datacenter and add them to
> > > the old cluster.
> > > 2. Use replica assignment tool to reassign all the partitions to
> brokers
> > > in new datacenter.
> > > 3. Perform controlled shutdown on the brokers in old datacenter.
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com>
> wrote:
> > >
> > > >Hi Team,
> > > >
> > > >in my project, we have built a new datacenter for Kafka brokers and
> > wants
> > > >to migrate from current datacenter to new one.
> > > >
> > > >Switching producers and consumers wont be a problem provided New
> > > >Datacenter
> > > >has all the messages of existing Datacenter.
> > > >
> > > >
> > > >i have only 1 topic with 2 partition that need to be migrated... that
> > too
> > > >it is only 1 time activity.
> > > >
> > > >Kindly suggest the best way to deal with this situation.
> > > >
> > > >
> > > >Regards,
> > > >Nitin Kumar Sharma.
> > >
> > >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
Regards,
Tao
Re: Kafka server relocation
Posted by nitin sharma <ku...@gmail.com>.
thanks Mayuresh and Jiangjie for your response.
I have actually not understood Mirror maker clearly and hence bit skeptical
if i will be able to execute it effectively.
Online i hv seen the following command to be execute, but not understood
what is consumer-1 & -2.properties here? do i need to copy from my
consumer code? also, any reason why to provide consumer property?
bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
consumer-1.properties --consumer.config consumer-2.properties
--producer.config producer.properties --whitelist my-topic
Regards,
Nitin Kumar Sharma.
On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> You can use the Mirror maker to move data from one data center to other and
> once all the data has been moved you can shut down the source data center
> by doing a controlled shutdown.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > If you want to do a seamless migration. I think a better way is to build
> a
> > cross datacenter Kafka cluster temporarily. So the process is:
> > 1. Add several new Kafka brokers in your new datacenter and add them to
> > the old cluster.
> > 2. Use replica assignment tool to reassign all the partitions to brokers
> > in new datacenter.
> > 3. Perform controlled shutdown on the brokers in old datacenter.
> >
> > Jiangjie (Becket) Qin
> >
> > On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com> wrote:
> >
> > >Hi Team,
> > >
> > >in my project, we have built a new datacenter for Kafka brokers and
> wants
> > >to migrate from current datacenter to new one.
> > >
> > >Switching producers and consumers wont be a problem provided New
> > >Datacenter
> > >has all the messages of existing Datacenter.
> > >
> > >
> > >i have only 1 topic with 2 partition that need to be migrated... that
> too
> > >it is only 1 time activity.
> > >
> > >Kindly suggest the best way to deal with this situation.
> > >
> > >
> > >Regards,
> > >Nitin Kumar Sharma.
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Kafka server relocation
Posted by Mayuresh Gharat <gh...@gmail.com>.
You can use the Mirror maker to move data from one data center to other and
once all the data has been moved you can shut down the source data center
by doing a controlled shutdown.
Thanks,
Mayuresh
On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> If you want to do a seamless migration. I think a better way is to build a
> cross datacenter Kafka cluster temporarily. So the process is:
> 1. Add several new Kafka brokers in your new datacenter and add them to
> the old cluster.
> 2. Use replica assignment tool to reassign all the partitions to brokers
> in new datacenter.
> 3. Perform controlled shutdown on the brokers in old datacenter.
>
> Jiangjie (Becket) Qin
>
> On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com> wrote:
>
> >Hi Team,
> >
> >in my project, we have built a new datacenter for Kafka brokers and wants
> >to migrate from current datacenter to new one.
> >
> >Switching producers and consumers wont be a problem provided New
> >Datacenter
> >has all the messages of existing Datacenter.
> >
> >
> >i have only 1 topic with 2 partition that need to be migrated... that too
> >it is only 1 time activity.
> >
> >Kindly suggest the best way to deal with this situation.
> >
> >
> >Regards,
> >Nitin Kumar Sharma.
>
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Kafka server relocation
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
If you want to do a seamless migration. I think a better way is to build a
cross datacenter Kafka cluster temporarily. So the process is:
1. Add several new Kafka brokers in your new datacenter and add them to
the old cluster.
2. Use replica assignment tool to reassign all the partitions to brokers
in new datacenter.
3. Perform controlled shutdown on the brokers in old datacenter.
Jiangjie (Becket) Qin
On 3/25/15, 2:01 PM, "nitin sharma" <ku...@gmail.com> wrote:
>Hi Team,
>
>in my project, we have built a new datacenter for Kafka brokers and wants
>to migrate from current datacenter to new one.
>
>Switching producers and consumers wont be a problem provided New
>Datacenter
>has all the messages of existing Datacenter.
>
>
>i have only 1 topic with 2 partition that need to be migrated... that too
>it is only 1 time activity.
>
>Kindly suggest the best way to deal with this situation.
>
>
>Regards,
>Nitin Kumar Sharma.