You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Muqtafi Akhmad <mu...@traveloka.com> on 2015/11/05 06:10:46 UTC
Kafka Out Of Memory Error
Hello all,
Recently I got incident with kafka cluster, I found OutOfMemoryError in
kafka server log
> WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest;
> Version: 0; CorrelationId: 7041716; ClientId: ReplicaFetcherThread-0-2;
> ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:[. . . . .].
> Possible cause: java.lang.OutOfMemoryError: Java heap space
> (kafka.server.ReplicaFetcherThread)
And at the same time there are socket timeout exceptions in kafka consumer.
Is there any hypothesis about what might be wrong?
Here is the cluster spec :
3 hosts, @ with memory 16 G
partitions : ~200
replication factor : 3
rate in : 300 K/s
rate out : 800 K/s
Thank you,
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Kafka Out Of Memory Error
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
My 3 node cluster (2x4 core xeon, 8GB ram per machine, striped HDD) had
crashed with OOM when I had some 5-6 topics with 256 partitions per topic.
dont remember the heap size I used, I think it was the default one that is
there in the 0.8.2.1 bundle
I have been trying to come up with the numbers around this, as to how many
topics and partitions can a cluster of n nodes support. I have used the
producer performance script to pump data into the cluster. But, that script
has some issues and some parameters that are passed form command line are
not exactly accepted.
I'll try and take out some time on it and give out the numbers
Regards,
Prabhjot
On Thu, Nov 5, 2015 at 2:58 PM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:
> Hello Prabhjot,
>
> Actually what I meant in previous email is that there are 200 topics, with
> 1 partition each so there are 200 total partitions. Is there any rule of
> thumb regarding this matter? Can you share your configuration (including
> spec, jvm memory, etc) for kafka cluster?
>
> Thank you,
>
> On Thu, Nov 5, 2015 at 4:07 PM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Not sure. But, I had hit OOM when using too many partitions and many
> topics
> > with a lesser heap size assigned to the jvm
> >
> > Regards,
> > Prabhjot
> >
> > On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad <mu...@traveloka.com>
> > wrote:
> >
> > > Hello all,
> > > Recently I got incident with kafka cluster, I found OutOfMemoryError in
> > > kafka server log
> > >
> > > > WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest;
> > > > Version: 0; CorrelationId: 7041716; ClientId:
> ReplicaFetcherThread-0-2;
> > > > ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:[. . .
> .
> > > .].
> > > > Possible cause: java.lang.OutOfMemoryError: Java heap space
> > > > (kafka.server.ReplicaFetcherThread)
> > >
> > > And at the same time there are socket timeout exceptions in kafka
> > consumer.
> > > Is there any hypothesis about what might be wrong?
> > >
> > > Here is the cluster spec :
> > > 3 hosts, @ with memory 16 G
> > > partitions : ~200
> > > replication factor : 3
> > > rate in : 300 K/s
> > > rate out : 800 K/s
> > >
> > > Thank you,
> > >
> > > --
> > > Muqtafi Akhmad
> > > Software Engineer
> > > Traveloka
> > >
> >
> >
> >
> > --
> > ---------------------------------------------------------
> > "There are only 10 types of people in the world: Those who understand
> > binary, and those who don't"
> >
>
>
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Kafka Out Of Memory Error
Posted by Muqtafi Akhmad <mu...@traveloka.com>.
Hello Prabhjot,
Actually what I meant in previous email is that there are 200 topics, with
1 partition each so there are 200 total partitions. Is there any rule of
thumb regarding this matter? Can you share your configuration (including
spec, jvm memory, etc) for kafka cluster?
Thank you,
On Thu, Nov 5, 2015 at 4:07 PM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi,
>
> Not sure. But, I had hit OOM when using too many partitions and many topics
> with a lesser heap size assigned to the jvm
>
> Regards,
> Prabhjot
>
> On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad <mu...@traveloka.com>
> wrote:
>
> > Hello all,
> > Recently I got incident with kafka cluster, I found OutOfMemoryError in
> > kafka server log
> >
> > > WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest;
> > > Version: 0; CorrelationId: 7041716; ClientId: ReplicaFetcherThread-0-2;
> > > ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:[. . . .
> > .].
> > > Possible cause: java.lang.OutOfMemoryError: Java heap space
> > > (kafka.server.ReplicaFetcherThread)
> >
> > And at the same time there are socket timeout exceptions in kafka
> consumer.
> > Is there any hypothesis about what might be wrong?
> >
> > Here is the cluster spec :
> > 3 hosts, @ with memory 16 G
> > partitions : ~200
> > replication factor : 3
> > rate in : 300 K/s
> > rate out : 800 K/s
> >
> > Thank you,
> >
> > --
> > Muqtafi Akhmad
> > Software Engineer
> > Traveloka
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Kafka Out Of Memory Error
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
Not sure. But, I had hit OOM when using too many partitions and many topics
with a lesser heap size assigned to the jvm
Regards,
Prabhjot
On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:
> Hello all,
> Recently I got incident with kafka cluster, I found OutOfMemoryError in
> kafka server log
>
> > WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest;
> > Version: 0; CorrelationId: 7041716; ClientId: ReplicaFetcherThread-0-2;
> > ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:[. . . .
> .].
> > Possible cause: java.lang.OutOfMemoryError: Java heap space
> > (kafka.server.ReplicaFetcherThread)
>
> And at the same time there are socket timeout exceptions in kafka consumer.
> Is there any hypothesis about what might be wrong?
>
> Here is the cluster spec :
> 3 hosts, @ with memory 16 G
> partitions : ~200
> replication factor : 3
> rate in : 300 K/s
> rate out : 800 K/s
>
> Thank you,
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"