You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Lance Laursen <ll...@rubiconproject.com> on 2015/05/01 02:34:32 UTC
Hitting integer limit when setting log segment.bytes
Hey all,
I am attempting to create a topic which uses 8GB log segment sizes, like so:
./kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest6p2r
--partitions 6 --replication-factor 2 --config max.message.bytes=655360
--config segment.bytes=8589934592
And am getting the following error:
Error while executing topic command For input string: "8589934592"
java.lang.NumberFormatException: For input string: "8589934592"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:583)
...
...
Upon further testing with --alter topic, it would appear that segment.bytes
will not accept a value higher than 2,147,483,647, which is the upper limit
for a signed 32bit int. This then restricts log segment size to an upper
limit of ~2GB.
We run Kafka on hard drive dense machines, each with 10gbit uplinks. We can
set ulimits higher in order to deal with all the open file handles (since
Kafka keeps all log segment file handles open), but it would be preferable
to minimize this number, as well as minimize the amount of log segment
rollover experienced at high traffic (ie: a rollover every 1-2 seconds or
so when saturating 10gbe).
Is there a reason (performance or otherwise) that a 32 bit integer is used
rather than something larger?
Thanks,
-Lance
Re: Hitting integer limit when setting log segment.bytes
Posted by Lance Laursen <ll...@rubiconproject.com>.
Great, thanks for the link Mike.
>From what I can tell, the only time opening of a segment file would be slow
is in the event of unclean shutdown, where a segment file may not have been
fsync'd and Kafka needs to CRC it and rebuild its index. This should really
only be a problem for the "newest" log segment and only with a limited
subset of topics which would be set to a high segment size.
However, having to necessitate a larger pointer (and accompanied increased
index size) just to accommodate large topics may be a good enough reason to
neglect large segments and embrace opening 20,000 files per restart, as it
seems the common use case for kafka involves a many-small-messages workload.
Thanks for the responses
On Wed, May 13, 2015 at 11:52 PM, Mike Axiak <mi...@axiak.net> wrote:
> Jay Kreps has commented on this before:
>
> https://issues.apache.org/jira/browse/KAFKA-1670?focusedCommentId=14161185&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14161185
>
> Basically, you can always have more segment files. Having too large of
> segment files will significantly slow down the opening of files which is
> done whenever a broker comes online or has to recover.
>
> On Thu, May 14, 2015 at 3:10 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> > wrote:
>
> > I suppose it is way log management works in kafka.
> > I am not sure the exact reason for this. Also the index files that are
> > constructed have a mapping of relative offset to the base offset of log
> > file to the real offset. The key value in index file is of the form
> > <Int,Int>.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > On Wed, May 13, 2015 at 5:57 PM, Lance Laursen <
> > llaursen@rubiconproject.com>
> > wrote:
> >
> > > Hey folks,
> > >
> > > Any update on this?
> > >
> > > On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen <
> > > llaursen@rubiconproject.com>
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > I am attempting to create a topic which uses 8GB log segment sizes,
> > like
> > > > so:
> > > > ./kafka-topics.sh --zookeeper localhost:2181 --create --topic
> > > perftest6p2r
> > > > --partitions 6 --replication-factor 2 --config
> max.message.bytes=655360
> > > > --config segment.bytes=8589934592
> > > >
> > > > And am getting the following error:
> > > > Error while executing topic command For input string: "8589934592"
> > > > java.lang.NumberFormatException: For input string: "8589934592"
> > > > at
> > > >
> > >
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > > > at java.lang.Integer.parseInt(Integer.java:583)
> > > > ...
> > > > ...
> > > >
> > > > Upon further testing with --alter topic, it would appear that
> > > > segment.bytes will not accept a value higher than 2,147,483,647,
> which
> > is
> > > > the upper limit for a signed 32bit int. This then restricts log
> segment
> > > > size to an upper limit of ~2GB.
> > > >
> > > > We run Kafka on hard drive dense machines, each with 10gbit uplinks.
> We
> > > > can set ulimits higher in order to deal with all the open file
> handles
> > > > (since Kafka keeps all log segment file handles open), but it would
> be
> > > > preferable to minimize this number, as well as minimize the amount of
> > log
> > > > segment rollover experienced at high traffic (ie: a rollover every
> 1-2
> > > > seconds or so when saturating 10gbe).
> > > >
> > > > Is there a reason (performance or otherwise) that a 32 bit integer is
> > > used
> > > > rather than something larger?
> > > >
> > > > Thanks,
> > > > -Lance
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > [image: elogo.png]
> > >
> > > Leading the Automation of Advertising
> > >
> > > LANCE LAURSEN | Systems Architect
> > >
> > > ••• (M) 310.903.0546
> > >
> > > 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094
> > >
> > > RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT
> > > <http://twitter.com/rubiconproject>
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
Re: Hitting integer limit when setting log segment.bytes
Posted by Mike Axiak <mi...@axiak.net>.
Jay Kreps has commented on this before:
https://issues.apache.org/jira/browse/KAFKA-1670?focusedCommentId=14161185&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14161185
Basically, you can always have more segment files. Having too large of
segment files will significantly slow down the opening of files which is
done whenever a broker comes online or has to recover.
On Thu, May 14, 2015 at 3:10 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> I suppose it is way log management works in kafka.
> I am not sure the exact reason for this. Also the index files that are
> constructed have a mapping of relative offset to the base offset of log
> file to the real offset. The key value in index file is of the form
> <Int,Int>.
>
>
> Thanks,
>
> Mayuresh
>
>
> On Wed, May 13, 2015 at 5:57 PM, Lance Laursen <
> llaursen@rubiconproject.com>
> wrote:
>
> > Hey folks,
> >
> > Any update on this?
> >
> > On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen <
> > llaursen@rubiconproject.com>
> > wrote:
> >
> > > Hey all,
> > >
> > > I am attempting to create a topic which uses 8GB log segment sizes,
> like
> > > so:
> > > ./kafka-topics.sh --zookeeper localhost:2181 --create --topic
> > perftest6p2r
> > > --partitions 6 --replication-factor 2 --config max.message.bytes=655360
> > > --config segment.bytes=8589934592
> > >
> > > And am getting the following error:
> > > Error while executing topic command For input string: "8589934592"
> > > java.lang.NumberFormatException: For input string: "8589934592"
> > > at
> > >
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > > at java.lang.Integer.parseInt(Integer.java:583)
> > > ...
> > > ...
> > >
> > > Upon further testing with --alter topic, it would appear that
> > > segment.bytes will not accept a value higher than 2,147,483,647, which
> is
> > > the upper limit for a signed 32bit int. This then restricts log segment
> > > size to an upper limit of ~2GB.
> > >
> > > We run Kafka on hard drive dense machines, each with 10gbit uplinks. We
> > > can set ulimits higher in order to deal with all the open file handles
> > > (since Kafka keeps all log segment file handles open), but it would be
> > > preferable to minimize this number, as well as minimize the amount of
> log
> > > segment rollover experienced at high traffic (ie: a rollover every 1-2
> > > seconds or so when saturating 10gbe).
> > >
> > > Is there a reason (performance or otherwise) that a 32 bit integer is
> > used
> > > rather than something larger?
> > >
> > > Thanks,
> > > -Lance
> > >
> > >
> >
> >
> > --
> >
> > [image: elogo.png]
> >
> > Leading the Automation of Advertising
> >
> > LANCE LAURSEN | Systems Architect
> >
> > ••• (M) 310.903.0546
> >
> > 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094
> >
> > RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT
> > <http://twitter.com/rubiconproject>
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Hitting integer limit when setting log segment.bytes
Posted by Mayuresh Gharat <gh...@gmail.com>.
I suppose it is way log management works in kafka.
I am not sure the exact reason for this. Also the index files that are
constructed have a mapping of relative offset to the base offset of log
file to the real offset. The key value in index file is of the form
<Int,Int>.
Thanks,
Mayuresh
On Wed, May 13, 2015 at 5:57 PM, Lance Laursen <ll...@rubiconproject.com>
wrote:
> Hey folks,
>
> Any update on this?
>
> On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen <
> llaursen@rubiconproject.com>
> wrote:
>
> > Hey all,
> >
> > I am attempting to create a topic which uses 8GB log segment sizes, like
> > so:
> > ./kafka-topics.sh --zookeeper localhost:2181 --create --topic
> perftest6p2r
> > --partitions 6 --replication-factor 2 --config max.message.bytes=655360
> > --config segment.bytes=8589934592
> >
> > And am getting the following error:
> > Error while executing topic command For input string: "8589934592"
> > java.lang.NumberFormatException: For input string: "8589934592"
> > at
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > at java.lang.Integer.parseInt(Integer.java:583)
> > ...
> > ...
> >
> > Upon further testing with --alter topic, it would appear that
> > segment.bytes will not accept a value higher than 2,147,483,647, which is
> > the upper limit for a signed 32bit int. This then restricts log segment
> > size to an upper limit of ~2GB.
> >
> > We run Kafka on hard drive dense machines, each with 10gbit uplinks. We
> > can set ulimits higher in order to deal with all the open file handles
> > (since Kafka keeps all log segment file handles open), but it would be
> > preferable to minimize this number, as well as minimize the amount of log
> > segment rollover experienced at high traffic (ie: a rollover every 1-2
> > seconds or so when saturating 10gbe).
> >
> > Is there a reason (performance or otherwise) that a 32 bit integer is
> used
> > rather than something larger?
> >
> > Thanks,
> > -Lance
> >
> >
>
>
> --
>
> [image: elogo.png]
>
> Leading the Automation of Advertising
>
> LANCE LAURSEN | Systems Architect
>
> ••• (M) 310.903.0546
>
> 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094
>
> RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT
> <http://twitter.com/rubiconproject>
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Hitting integer limit when setting log segment.bytes
Posted by Lance Laursen <ll...@rubiconproject.com>.
Hey folks,
Any update on this?
On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen <ll...@rubiconproject.com>
wrote:
> Hey all,
>
> I am attempting to create a topic which uses 8GB log segment sizes, like
> so:
> ./kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest6p2r
> --partitions 6 --replication-factor 2 --config max.message.bytes=655360
> --config segment.bytes=8589934592
>
> And am getting the following error:
> Error while executing topic command For input string: "8589934592"
> java.lang.NumberFormatException: For input string: "8589934592"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:583)
> ...
> ...
>
> Upon further testing with --alter topic, it would appear that
> segment.bytes will not accept a value higher than 2,147,483,647, which is
> the upper limit for a signed 32bit int. This then restricts log segment
> size to an upper limit of ~2GB.
>
> We run Kafka on hard drive dense machines, each with 10gbit uplinks. We
> can set ulimits higher in order to deal with all the open file handles
> (since Kafka keeps all log segment file handles open), but it would be
> preferable to minimize this number, as well as minimize the amount of log
> segment rollover experienced at high traffic (ie: a rollover every 1-2
> seconds or so when saturating 10gbe).
>
> Is there a reason (performance or otherwise) that a 32 bit integer is used
> rather than something larger?
>
> Thanks,
> -Lance
>
>
--
[image: elogo.png]
Leading the Automation of Advertising
LANCE LAURSEN | Systems Architect
••• (M) 310.903.0546
12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094
RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT
<http://twitter.com/rubiconproject>