You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kunal Gupta <ku...@cube26.com> on 2016/03/14 07:10:03 UTC
Need a help in understanding __consumer_offsets topic creation in
Kafka Cluster
Hi everyone,
I am new here, recently join the group. I faced a problem in Kafka Cluster,
a problem is described below.
I am using Kafka version 0.9.0.0
We have established a Kafka Cluster of 3 machines where 2 machines are
utilized for Kafka broker and same 3 machines utilized for zookeeper. When
we start our consumer written in Kafka-Python, Kafka brokers create a new
topic name __consumer_offsets with 50 partitions and spread over 100 % in
Kafka broker with replica 2.
I know why topic is get created it is because offsets are stored in Kafka
broker instead of zookeeper but problem is each partition of
__consumer_offsets take a lot of space as compared to topic data and
yesterday I have restarted my cluster gain due to lack of space.
Can you exactly help me why __consumer_offsets are taking a lot of space?
and how would I get rid of this.
Topic data space is 2 GB and while __consumer_offsets are taking 92 GB. My
server size is 98 GB with 4 GB RAM.
*Thanks, Kunal*
*+91-9958189589*
*Data Analyst*
*First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
<http://dl.acm.org/citation.cfm?id=2790798> *
*Blog:- **http://learnhardwithkunalgupta.blogspot.in
<http://learnhardwithkunalgupta.blogspot.in> *
Re: Need a help in understanding __consumer_offsets topic creation in
Kafka Cluster
Posted by Achanta Vamsi Subhash <ac...@flipkart.com>.
We changed the policy to "delete" dynamically for the __consumer_offsets
topic and it was a better option than doing a cluster restart after
enabling log compaction. Also, we found problems when you are replicating
to a log compacted topic from a non-compacted topic (which is leader).
On Mon, Mar 14, 2016 at 12:07 PM, Kunal Gupta <ku...@cube26.com>
wrote:
> Thanks @Stevo Slavić
>
>
> *Thanks, Kunal*
> *+91-9958189589*
> *Data Analyst*
> *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> <http://dl.acm.org/citation.cfm?id=2790798> *
> *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> <http://learnhardwithkunalgupta.blogspot.in> *
>
>
>
>
>
> On Mon, Mar 14, 2016 at 11:57 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > You are affected by this 0.9.0.0 bug
> > https://issues.apache.org/jira/browse/KAFKA-2988
> >
> > It was fixed for 0.9.0.1.
> >
> > You could just apply same fix to your 0.9.0.0 cluster but I'd recommend
> > upgrading to 0.9.0.1.
> >
> > Kind regards,
> > Stevo Slavic.
> > On Mon, Mar 14, 2016, 07:10 Kunal Gupta <ku...@cube26.com> wrote:
> >
> > > Hi everyone,
> > >
> > > I am new here, recently join the group. I faced a problem in Kafka
> > Cluster,
> > > a problem is described below.
> > >
> > > I am using Kafka version 0.9.0.0
> > >
> > > We have established a Kafka Cluster of 3 machines where 2 machines are
> > > utilized for Kafka broker and same 3 machines utilized for zookeeper.
> > When
> > > we start our consumer written in Kafka-Python, Kafka brokers create a
> new
> > > topic name __consumer_offsets with 50 partitions and spread over 100 %
> in
> > > Kafka broker with replica 2.
> > >
> > > I know why topic is get created it is because offsets are stored in
> Kafka
> > > broker instead of zookeeper but problem is each partition of
> > > __consumer_offsets take a lot of space as compared to topic data and
> > > yesterday I have restarted my cluster gain due to lack of space.
> > >
> > > Can you exactly help me why __consumer_offsets are taking a lot of
> > space?
> > > and how would I get rid of this.
> > >
> > > Topic data space is 2 GB and while __consumer_offsets are taking 92 GB.
> > My
> > > server size is 98 GB with 4 GB RAM.
> > >
> > >
> > >
> > >
> > > *Thanks, Kunal*
> > > *+91-9958189589*
> > > *Data Analyst*
> > > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> > > <http://dl.acm.org/citation.cfm?id=2790798> *
> > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > > <http://learnhardwithkunalgupta.blogspot.in> *
> > >
> >
>
--
Regards
Vamsi Subhash
Re: Need a help in understanding __consumer_offsets topic creation in
Kafka Cluster
Posted by Kunal Gupta <ku...@cube26.com>.
Thanks @Stevo Slavić
*Thanks, Kunal*
*+91-9958189589*
*Data Analyst*
*First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
<http://dl.acm.org/citation.cfm?id=2790798> *
*Blog:- **http://learnhardwithkunalgupta.blogspot.in
<http://learnhardwithkunalgupta.blogspot.in> *
On Mon, Mar 14, 2016 at 11:57 AM, Stevo Slavić <ss...@gmail.com> wrote:
> You are affected by this 0.9.0.0 bug
> https://issues.apache.org/jira/browse/KAFKA-2988
>
> It was fixed for 0.9.0.1.
>
> You could just apply same fix to your 0.9.0.0 cluster but I'd recommend
> upgrading to 0.9.0.1.
>
> Kind regards,
> Stevo Slavic.
> On Mon, Mar 14, 2016, 07:10 Kunal Gupta <ku...@cube26.com> wrote:
>
> > Hi everyone,
> >
> > I am new here, recently join the group. I faced a problem in Kafka
> Cluster,
> > a problem is described below.
> >
> > I am using Kafka version 0.9.0.0
> >
> > We have established a Kafka Cluster of 3 machines where 2 machines are
> > utilized for Kafka broker and same 3 machines utilized for zookeeper.
> When
> > we start our consumer written in Kafka-Python, Kafka brokers create a new
> > topic name __consumer_offsets with 50 partitions and spread over 100 % in
> > Kafka broker with replica 2.
> >
> > I know why topic is get created it is because offsets are stored in Kafka
> > broker instead of zookeeper but problem is each partition of
> > __consumer_offsets take a lot of space as compared to topic data and
> > yesterday I have restarted my cluster gain due to lack of space.
> >
> > Can you exactly help me why __consumer_offsets are taking a lot of
> space?
> > and how would I get rid of this.
> >
> > Topic data space is 2 GB and while __consumer_offsets are taking 92 GB.
> My
> > server size is 98 GB with 4 GB RAM.
> >
> >
> >
> >
> > *Thanks, Kunal*
> > *+91-9958189589*
> > *Data Analyst*
> > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> > <http://dl.acm.org/citation.cfm?id=2790798> *
> > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > <http://learnhardwithkunalgupta.blogspot.in> *
> >
>
Re: Need a help in understanding __consumer_offsets topic creation in
Kafka Cluster
Posted by Stevo Slavić <ss...@gmail.com>.
You are affected by this 0.9.0.0 bug
https://issues.apache.org/jira/browse/KAFKA-2988
It was fixed for 0.9.0.1.
You could just apply same fix to your 0.9.0.0 cluster but I'd recommend
upgrading to 0.9.0.1.
Kind regards,
Stevo Slavic.
On Mon, Mar 14, 2016, 07:10 Kunal Gupta <ku...@cube26.com> wrote:
> Hi everyone,
>
> I am new here, recently join the group. I faced a problem in Kafka Cluster,
> a problem is described below.
>
> I am using Kafka version 0.9.0.0
>
> We have established a Kafka Cluster of 3 machines where 2 machines are
> utilized for Kafka broker and same 3 machines utilized for zookeeper. When
> we start our consumer written in Kafka-Python, Kafka brokers create a new
> topic name __consumer_offsets with 50 partitions and spread over 100 % in
> Kafka broker with replica 2.
>
> I know why topic is get created it is because offsets are stored in Kafka
> broker instead of zookeeper but problem is each partition of
> __consumer_offsets take a lot of space as compared to topic data and
> yesterday I have restarted my cluster gain due to lack of space.
>
> Can you exactly help me why __consumer_offsets are taking a lot of space?
> and how would I get rid of this.
>
> Topic data space is 2 GB and while __consumer_offsets are taking 92 GB. My
> server size is 98 GB with 4 GB RAM.
>
>
>
>
> *Thanks, Kunal*
> *+91-9958189589*
> *Data Analyst*
> *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> <http://dl.acm.org/citation.cfm?id=2790798> *
> *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> <http://learnhardwithkunalgupta.blogspot.in> *
>