You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Guozhang Wang <wa...@gmail.com> on 2014/06/01 01:05:09 UTC
Re: kafka-reassign-partitions.sh leaves data on source broker?
Hello Otis,
What is your replication factor?
Guozhang
On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to
> balance them (read: free disk space on the overloaded broker).
>
> Is there a tool that will just rebalance everything? I couldn't find one,
> so I ran this:
>
> $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> --execute
>
> This seems to have *copied* about half of the partitions of the topic
> specified in the given JSON file from broker 2 (source) to broker 1
> (target). Good.
>
> I see this in broker 2 (source broker):
>
> [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2] Removed
> fetcher for partitions [spm_new_cluster_free_system_topic,1]
> (kafka.server.ReplicaFetcherManager)
>
> ... which looks promising.... but:
>
> My broker 2 (source) *still* has all partitions of the problematic topic.
> I thought it would remove the moved partitions from the source broker....
> no?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
--
-- Guozhang
Re: kafka-reassign-partitions.sh leaves data on source broker?
Posted by Guozhang Wang <wa...@gmail.com>.
Json file should be correct, and I have just checked the code, it should
send the stop replica request to the old leaders to delete their logs. If
you can consistently re-produce this issue could you file a jira?
Guozhang
On Sat, May 31, 2014 at 7:40 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> No, I don't think I saw anything with "Stop" in the logs, but I may have
> lost those old logs, so I'm not 100% sure.
>
> I just re-ran the same command and I didn't see anything with "Stop" in
> broker 2's (source) log.
>
> I did see log messages like the following one in broker 2's logs as soon as
> I ran this:
>
> $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> --execute
>
> [2014-06-01 02:29:31,256] ERROR [KafkaApi-2] Error when processing fetch
> request for partition [spm_new_cluster_free_system_topic,4] offset 0 from
> follower with correlation id 1 (kafka.server.KafkaApis)
> kafka.common.OffsetOutOfRangeException: Request for offset 0 but we only
> have log segments in the range 16400532 to 35860153.
> at kafka.log.Log.read(Log.scala:380)
> at
>
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530)
> at
>
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476)
> at
>
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471)
>
> ... but I'm not sure what to make of it. The /tmp/partition.json file was
> generated with --generate and contains:
>
> {"version":1,"partitions":[
>
> {"topic":"spm_new_cluster_free_system_topic","partition":0,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":6,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":11,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":4,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":2,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":9,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":3,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":5,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":7,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":1,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":10,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":8,"replicas":[1]}]}
>
> Does the above mean, in plain English, "move partitions 0-11 for
> topic spm_new_cluster_free_system_topic to broker 1"?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sat, May 31, 2014 at 8:32 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > You may take a look at the broker configs for replication factor:
> >
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > With replication factor 1 (default value) the broker holding the old
> leader
> > should received StopReplica Request to stop the replica and delete the
> > segment files. Do you see this log entries on broker 2's request logs?
> >
> > Guozhang
> >
> >
> > On Sat, May 31, 2014 at 5:15 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi Guozhang,
> > >
> > > I don't have any replication-related properties in my config file and I
> > > don't see any mentions of replication at startup in the log, so I don't
> > > really know.
> > >
> > > The only time I saw replication mentioned in the log was during some
> > errors
> > > that said my replication factor was 1 and that was > the number of
> > brokers.
> > > So based on this error message I'd say my replication factor is 1.
> > >
> > > Is there a replication property I could explicitly set?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Performance Monitoring * Log Analytics * Search Analytics
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hello Otis,
> > > >
> > > > What is your replication factor?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying
> to
> > > > > balance them (read: free disk space on the overloaded broker).
> > > > >
> > > > > Is there a tool that will just rebalance everything? I couldn't
> find
> > > > one,
> > > > > so I ran this:
> > > > >
> > > > > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > > > > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > > > > --execute
> > > > >
> > > > > This seems to have *copied* about half of the partitions of the
> topic
> > > > > specified in the given JSON file from broker 2 (source) to broker 1
> > > > > (target). Good.
> > > > >
> > > > > I see this in broker 2 (source broker):
> > > > >
> > > > > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> > > > Removed
> > > > > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > > > > (kafka.server.ReplicaFetcherManager)
> > > > >
> > > > > ... which looks promising.... but:
> > > > >
> > > > > My broker 2 (source) *still* has all partitions of the problematic
> > > topic.
> > > > > I thought it would remove the moved partitions from the source
> > > broker....
> > > > > no?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > Performance Monitoring * Log Analytics * Search Analytics
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka-reassign-partitions.sh leaves data on source broker?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
No, I don't think I saw anything with "Stop" in the logs, but I may have
lost those old logs, so I'm not 100% sure.
I just re-ran the same command and I didn't see anything with "Stop" in
broker 2's (source) log.
I did see log messages like the following one in broker 2's logs as soon as
I ran this:
$ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
--execute
[2014-06-01 02:29:31,256] ERROR [KafkaApi-2] Error when processing fetch
request for partition [spm_new_cluster_free_system_topic,4] offset 0 from
follower with correlation id 1 (kafka.server.KafkaApis)
kafka.common.OffsetOutOfRangeException: Request for offset 0 but we only
have log segments in the range 16400532 to 35860153.
at kafka.log.Log.read(Log.scala:380)
at
kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530)
at
kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476)
at
kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471)
... but I'm not sure what to make of it. The /tmp/partition.json file was
generated with --generate and contains:
{"version":1,"partitions":[
{"topic":"spm_new_cluster_free_system_topic","partition":0,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":6,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":11,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":4,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":2,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":9,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":3,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":5,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":7,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":1,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":10,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":8,"replicas":[1]}]}
Does the above mean, in plain English, "move partitions 0-11 for
topic spm_new_cluster_free_system_topic to broker 1"?
Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, May 31, 2014 at 8:32 PM, Guozhang Wang <wa...@gmail.com> wrote:
> You may take a look at the broker configs for replication factor:
>
> http://kafka.apache.org/documentation.html#brokerconfigs
>
> With replication factor 1 (default value) the broker holding the old leader
> should received StopReplica Request to stop the replica and delete the
> segment files. Do you see this log entries on broker 2's request logs?
>
> Guozhang
>
>
> On Sat, May 31, 2014 at 5:15 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi Guozhang,
> >
> > I don't have any replication-related properties in my config file and I
> > don't see any mentions of replication at startup in the log, so I don't
> > really know.
> >
> > The only time I saw replication mentioned in the log was during some
> errors
> > that said my replication factor was 1 and that was > the number of
> brokers.
> > So based on this error message I'd say my replication factor is 1.
> >
> > Is there a replication property I could explicitly set?
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hello Otis,
> > >
> > > What is your replication factor?
> > >
> > > Guozhang
> > >
> > >
> > > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to
> > > > balance them (read: free disk space on the overloaded broker).
> > > >
> > > > Is there a tool that will just rebalance everything? I couldn't find
> > > one,
> > > > so I ran this:
> > > >
> > > > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > > > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > > > --execute
> > > >
> > > > This seems to have *copied* about half of the partitions of the topic
> > > > specified in the given JSON file from broker 2 (source) to broker 1
> > > > (target). Good.
> > > >
> > > > I see this in broker 2 (source broker):
> > > >
> > > > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> > > Removed
> > > > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > > > (kafka.server.ReplicaFetcherManager)
> > > >
> > > > ... which looks promising.... but:
> > > >
> > > > My broker 2 (source) *still* has all partitions of the problematic
> > topic.
> > > > I thought it would remove the moved partitions from the source
> > broker....
> > > > no?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > Performance Monitoring * Log Analytics * Search Analytics
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka-reassign-partitions.sh leaves data on source broker?
Posted by Guozhang Wang <wa...@gmail.com>.
You may take a look at the broker configs for replication factor:
http://kafka.apache.org/documentation.html#brokerconfigs
With replication factor 1 (default value) the broker holding the old leader
should received StopReplica Request to stop the replica and delete the
segment files. Do you see this log entries on broker 2's request logs?
Guozhang
On Sat, May 31, 2014 at 5:15 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi Guozhang,
>
> I don't have any replication-related properties in my config file and I
> don't see any mentions of replication at startup in the log, so I don't
> really know.
>
> The only time I saw replication mentioned in the log was during some errors
> that said my replication factor was 1 and that was > the number of brokers.
> So based on this error message I'd say my replication factor is 1.
>
> Is there a replication property I could explicitly set?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hello Otis,
> >
> > What is your replication factor?
> >
> > Guozhang
> >
> >
> > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to
> > > balance them (read: free disk space on the overloaded broker).
> > >
> > > Is there a tool that will just rebalance everything? I couldn't find
> > one,
> > > so I ran this:
> > >
> > > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > > --execute
> > >
> > > This seems to have *copied* about half of the partitions of the topic
> > > specified in the given JSON file from broker 2 (source) to broker 1
> > > (target). Good.
> > >
> > > I see this in broker 2 (source broker):
> > >
> > > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> > Removed
> > > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > > (kafka.server.ReplicaFetcherManager)
> > >
> > > ... which looks promising.... but:
> > >
> > > My broker 2 (source) *still* has all partitions of the problematic
> topic.
> > > I thought it would remove the moved partitions from the source
> broker....
> > > no?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Performance Monitoring * Log Analytics * Search Analytics
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka-reassign-partitions.sh leaves data on source broker?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Guozhang,
I don't have any replication-related properties in my config file and I
don't see any mentions of replication at startup in the log, so I don't
really know.
The only time I saw replication mentioned in the log was during some errors
that said my replication factor was 1 and that was > the number of brokers.
So based on this error message I'd say my replication factor is 1.
Is there a replication property I could explicitly set?
Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Otis,
>
> What is your replication factor?
>
> Guozhang
>
>
> On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to
> > balance them (read: free disk space on the overloaded broker).
> >
> > Is there a tool that will just rebalance everything? I couldn't find
> one,
> > so I ran this:
> >
> > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > --execute
> >
> > This seems to have *copied* about half of the partitions of the topic
> > specified in the given JSON file from broker 2 (source) to broker 1
> > (target). Good.
> >
> > I see this in broker 2 (source broker):
> >
> > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> Removed
> > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > (kafka.server.ReplicaFetcherManager)
> >
> > ... which looks promising.... but:
> >
> > My broker 2 (source) *still* has all partitions of the problematic topic.
> > I thought it would remove the moved partitions from the source broker....
> > no?
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
>
>
>
> --
> -- Guozhang
>