You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Enrico Olivelli <eo...@gmail.com> on 2018/03/05 11:44:03 UTC
Broker cannot start switch to Java9 - weird file system issue ?
Hi,
This is a very strage case. I have a Kafka broker (part of a cluster of 3
brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
9.0.4.
There are a lot of .index and .timeindex files taking 10MB, they are for
empty partiions.
Running with Java 9 the server seems to rebuild these files and each file
takes "really" 10MB.
The sum of all the files (calculated using du -sh) is 22GB and the broker
crashes during startup, disk becomes full and no log more is written. (I
can send an extraction of the logs, but the tell only about 'rebuilding
index', the same as on Java 8)
Reverting the same broker to Java 8 and removing the index files, the
broker rebuilds such files, each files take 10MB, but the full sum of sizes
(calculated using du -sh) is 38 MB !
I am running this broker on CentosOS 7 on EXT4 FS.
I have upgraded the broker to latest and greatest Kafka 1.0.0 (from 0.10.2)
without any success.
All of the other testing clusters on CentOS7 (same SO settings) did not
have any problem.
The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8
it works fine with 1GB and default max direct memory size)
Which are the relevant configuration options ?
Thoughts ?
Thank you
Enrico
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks, I updated it.
On Thu, Apr 26, 2018 at 4:09 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Here it is !
>
> https://issues.apache.org/jira/browse/KAFKA-6828
>
> Thank you
>
> Enrico
>
> 2018-04-24 20:53 GMT+02:00 Ismael Juma <is...@juma.me.uk>:
>
> > A JIRA ticket would be appreciated. :)
> >
> > Ismael
> >
> > On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Il sab 21 apr 2018, 06:29 Ismael Juma <is...@juma.me.uk> ha scritto:
> > >
> > > > Hi Enrico,
> > > >
> > > > It is a real problem because it causes indexes to take a lot more
> disk
> > > > space upfront. The sparsity is an important if people over partition,
> > for
> > > > example.
> > > >
> > >
> > > Got it.
> > > In production I saw no issue, maybe due to much availability of disk
> > space.
> > >
> > > Should I file a JIRA or you will do?
> > > Enrico
> > >
> > >
> > > > Ismael
> > > >
> > > > On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <
> eolivelli@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha
> > scritto:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > Coincidentally, I saw your message to nio-dev and followed up
> > there.
> > > > > >
> > > > >
> > > > > I think this is not a 'real' problem, you will notice the
> difference
> > > only
> > > > > if you have a lot of empty topics/partitions.
> > > > > In fact when you simply upgrade to jdk9/10 immediately you are
> > charged
> > > > with
> > > > > 10MB of disk space for each partition.
> > > > > In production I did not suffer this change because usually you do
> not
> > > > have
> > > > > empty partitions.
> > > > > In my test environments, where I had thousands of test empty
> > > partitions,
> > > > > disks filled up immediately and without any reason, it took time to
> > > > > understand the real cause. Broker will crash without much 'log' as
> > disk
> > > > is
> > > > > out of space.
> > > > >
> > > > > Maybe it would be useful to add some notice about this potential
> > > problem
> > > > > during the upgrade of the jdk.
> > > > >
> > > > > Hope that helps
> > > > > Enrico
> > > > >
> > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <
> > > eolivelli@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > It is a deliberate change in JDK code
> > > > > > >
> > > > > > > Just for reference see this discussion on nio-dev list on
> > OpenJDK
> > > > > > >
> > > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/
> 005008.html
> > > > > > >
> > > > > > >
> > > > > > > see
> > > > > > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > > > > > >
> > > > > > > Cheers
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <
> eolivelli@gmail.com
> > >:
> > > > > > >
> > > > > > > > Workaround:
> > > > > > > > as these brokers are only for test environments I have set
> very
> > > > small
> > > > > > > > values for index file size, which affects pre-allocation
> > > > > > > > segment.index.bytes=65536
> > > > > > > > log.index.size.max.bytes=65536
> > > > > > > >
> > > > > > > > If anyone has some thought it will be very appreciated
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > Enrico
> > > > > > > >
> > > > > > > >
> > > > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <
> > eolivelli@gmail.com
> > > >:
> > > > > > > >
> > > > > > > >> The only fact I have found is that with Java8 Kafka is
> > creating
> > > > > > "SPARSE"
> > > > > > > >> files and with Java9 this is not true anymore
> > > > > > > >>
> > > > > > > >> Enrico
> > > > > > > >>
> > > > > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <
> > eolivelli@gmail.com
> > > >:
> > > > > > > >>
> > > > > > > >>> Hi,
> > > > > > > >>> This is a very strage case. I have a Kafka broker (part of
> a
> > > > > cluster
> > > > > > of
> > > > > > > >>> 3 brokers) which cannot start upgrading Java from Oracle
> JDK8
> > > to
> > > > > > > Oracle JDK
> > > > > > > >>> 9.0.4.
> > > > > > > >>>
> > > > > > > >>> There are a lot of .index and .timeindex files taking 10MB,
> > > they
> > > > > are
> > > > > > > for
> > > > > > > >>> empty partiions.
> > > > > > > >>>
> > > > > > > >>> Running with Java 9 the server seems to rebuild these files
> > and
> > > > > each
> > > > > > > >>> file takes "really" 10MB.
> > > > > > > >>> The sum of all the files (calculated using du -sh) is 22GB
> > and
> > > > the
> > > > > > > >>> broker crashes during startup, disk becomes full and no log
> > > more
> > > > is
> > > > > > > >>> written. (I can send an extraction of the logs, but the
> tell
> > > only
> > > > > > > about
> > > > > > > >>> 'rebuilding index', the same as on Java 8)
> > > > > > > >>>
> > > > > > > >>> Reverting the same broker to Java 8 and removing the index
> > > files,
> > > > > the
> > > > > > > >>> broker rebuilds such files, each files take 10MB, but the
> > full
> > > > sum
> > > > > of
> > > > > > > sizes
> > > > > > > >>> (calculated using du -sh) is 38 MB !
> > > > > > > >>>
> > > > > > > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > > > > > > >>>
> > > > > > > >>> I have upgraded the broker to latest and greatest Kafka
> 1.0.0
> > > > (from
> > > > > > > >>> 0.10.2) without any success.
> > > > > > > >>>
> > > > > > > >>> All of the other testing clusters on CentOS7 (same SO
> > settings)
> > > > did
> > > > > > not
> > > > > > > >>> have any problem.
> > > > > > > >>>
> > > > > > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory
> > Size
> > > > (on
> > > > > > > >>> Java8 it works fine with 1GB and default max direct memory
> > > size)
> > > > > > > >>>
> > > > > > > >>> Which are the relevant configuration options ?
> > > > > > > >>>
> > > > > > > >>> Thoughts ?
> > > > > > > >>>
> > > > > > > >>> Thank you
> > > > > > > >>>
> > > > > > > >>> Enrico
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > >
> > > > > -- Enrico Olivelli
> > > > >
> > > >
> > > --
> > >
> > >
> > > -- Enrico Olivelli
> > >
> >
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
Here it is !
https://issues.apache.org/jira/browse/KAFKA-6828
Thank you
Enrico
2018-04-24 20:53 GMT+02:00 Ismael Juma <is...@juma.me.uk>:
> A JIRA ticket would be appreciated. :)
>
> Ismael
>
> On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il sab 21 apr 2018, 06:29 Ismael Juma <is...@juma.me.uk> ha scritto:
> >
> > > Hi Enrico,
> > >
> > > It is a real problem because it causes indexes to take a lot more disk
> > > space upfront. The sparsity is an important if people over partition,
> for
> > > example.
> > >
> >
> > Got it.
> > In production I saw no issue, maybe due to much availability of disk
> space.
> >
> > Should I file a JIRA or you will do?
> > Enrico
> >
> >
> > > Ismael
> > >
> > > On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <eolivelli@gmail.com
> >
> > > wrote:
> > >
> > > > Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha
> scritto:
> > > >
> > > > > Hi Enrico,
> > > > >
> > > > > Coincidentally, I saw your message to nio-dev and followed up
> there.
> > > > >
> > > >
> > > > I think this is not a 'real' problem, you will notice the difference
> > only
> > > > if you have a lot of empty topics/partitions.
> > > > In fact when you simply upgrade to jdk9/10 immediately you are
> charged
> > > with
> > > > 10MB of disk space for each partition.
> > > > In production I did not suffer this change because usually you do not
> > > have
> > > > empty partitions.
> > > > In my test environments, where I had thousands of test empty
> > partitions,
> > > > disks filled up immediately and without any reason, it took time to
> > > > understand the real cause. Broker will crash without much 'log' as
> disk
> > > is
> > > > out of space.
> > > >
> > > > Maybe it would be useful to add some notice about this potential
> > problem
> > > > during the upgrade of the jdk.
> > > >
> > > > Hope that helps
> > > > Enrico
> > > >
> > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <
> > eolivelli@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It is a deliberate change in JDK code
> > > > > >
> > > > > > Just for reference see this discussion on nio-dev list on
> OpenJDK
> > > > > >
> > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
> > > > > >
> > > > > >
> > > > > > see
> > > > > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > > > > >
> > > > > > Cheers
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com
> >:
> > > > > >
> > > > > > > Workaround:
> > > > > > > as these brokers are only for test environments I have set very
> > > small
> > > > > > > values for index file size, which affects pre-allocation
> > > > > > > segment.index.bytes=65536
> > > > > > > log.index.size.max.bytes=65536
> > > > > > >
> > > > > > > If anyone has some thought it will be very appreciated
> > > > > > > Cheers
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <
> eolivelli@gmail.com
> > >:
> > > > > > >
> > > > > > >> The only fact I have found is that with Java8 Kafka is
> creating
> > > > > "SPARSE"
> > > > > > >> files and with Java9 this is not true anymore
> > > > > > >>
> > > > > > >> Enrico
> > > > > > >>
> > > > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <
> eolivelli@gmail.com
> > >:
> > > > > > >>
> > > > > > >>> Hi,
> > > > > > >>> This is a very strage case. I have a Kafka broker (part of a
> > > > cluster
> > > > > of
> > > > > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8
> > to
> > > > > > Oracle JDK
> > > > > > >>> 9.0.4.
> > > > > > >>>
> > > > > > >>> There are a lot of .index and .timeindex files taking 10MB,
> > they
> > > > are
> > > > > > for
> > > > > > >>> empty partiions.
> > > > > > >>>
> > > > > > >>> Running with Java 9 the server seems to rebuild these files
> and
> > > > each
> > > > > > >>> file takes "really" 10MB.
> > > > > > >>> The sum of all the files (calculated using du -sh) is 22GB
> and
> > > the
> > > > > > >>> broker crashes during startup, disk becomes full and no log
> > more
> > > is
> > > > > > >>> written. (I can send an extraction of the logs, but the tell
> > only
> > > > > > about
> > > > > > >>> 'rebuilding index', the same as on Java 8)
> > > > > > >>>
> > > > > > >>> Reverting the same broker to Java 8 and removing the index
> > files,
> > > > the
> > > > > > >>> broker rebuilds such files, each files take 10MB, but the
> full
> > > sum
> > > > of
> > > > > > sizes
> > > > > > >>> (calculated using du -sh) is 38 MB !
> > > > > > >>>
> > > > > > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > > > > > >>>
> > > > > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0
> > > (from
> > > > > > >>> 0.10.2) without any success.
> > > > > > >>>
> > > > > > >>> All of the other testing clusters on CentOS7 (same SO
> settings)
> > > did
> > > > > not
> > > > > > >>> have any problem.
> > > > > > >>>
> > > > > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory
> Size
> > > (on
> > > > > > >>> Java8 it works fine with 1GB and default max direct memory
> > size)
> > > > > > >>>
> > > > > > >>> Which are the relevant configuration options ?
> > > > > > >>>
> > > > > > >>> Thoughts ?
> > > > > > >>>
> > > > > > >>> Thank you
> > > > > > >>>
> > > > > > >>> Enrico
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > >
> > > > -- Enrico Olivelli
> > > >
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Ismael Juma <is...@juma.me.uk>.
A JIRA ticket would be appreciated. :)
Ismael
On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Il sab 21 apr 2018, 06:29 Ismael Juma <is...@juma.me.uk> ha scritto:
>
> > Hi Enrico,
> >
> > It is a real problem because it causes indexes to take a lot more disk
> > space upfront. The sparsity is an important if people over partition, for
> > example.
> >
>
> Got it.
> In production I saw no issue, maybe due to much availability of disk space.
>
> Should I file a JIRA or you will do?
> Enrico
>
>
> > Ismael
> >
> > On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha scritto:
> > >
> > > > Hi Enrico,
> > > >
> > > > Coincidentally, I saw your message to nio-dev and followed up there.
> > > >
> > >
> > > I think this is not a 'real' problem, you will notice the difference
> only
> > > if you have a lot of empty topics/partitions.
> > > In fact when you simply upgrade to jdk9/10 immediately you are charged
> > with
> > > 10MB of disk space for each partition.
> > > In production I did not suffer this change because usually you do not
> > have
> > > empty partitions.
> > > In my test environments, where I had thousands of test empty
> partitions,
> > > disks filled up immediately and without any reason, it took time to
> > > understand the real cause. Broker will crash without much 'log' as disk
> > is
> > > out of space.
> > >
> > > Maybe it would be useful to add some notice about this potential
> problem
> > > during the upgrade of the jdk.
> > >
> > > Hope that helps
> > > Enrico
> > >
> > >
> > > > Ismael
> > > >
> > > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <
> eolivelli@gmail.com>
> > > > wrote:
> > > >
> > > > > It is a deliberate change in JDK code
> > > > >
> > > > > Just for reference see this discussion on nio-dev list on OpenJDK
> > > > >
> > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
> > > > >
> > > > >
> > > > > see
> > > > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > > > >
> > > > > Cheers
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > > >
> > > > > > Workaround:
> > > > > > as these brokers are only for test environments I have set very
> > small
> > > > > > values for index file size, which affects pre-allocation
> > > > > > segment.index.bytes=65536
> > > > > > log.index.size.max.bytes=65536
> > > > > >
> > > > > > If anyone has some thought it will be very appreciated
> > > > > > Cheers
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com
> >:
> > > > > >
> > > > > >> The only fact I have found is that with Java8 Kafka is creating
> > > > "SPARSE"
> > > > > >> files and with Java9 this is not true anymore
> > > > > >>
> > > > > >> Enrico
> > > > > >>
> > > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com
> >:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>> This is a very strage case. I have a Kafka broker (part of a
> > > cluster
> > > > of
> > > > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8
> to
> > > > > Oracle JDK
> > > > > >>> 9.0.4.
> > > > > >>>
> > > > > >>> There are a lot of .index and .timeindex files taking 10MB,
> they
> > > are
> > > > > for
> > > > > >>> empty partiions.
> > > > > >>>
> > > > > >>> Running with Java 9 the server seems to rebuild these files and
> > > each
> > > > > >>> file takes "really" 10MB.
> > > > > >>> The sum of all the files (calculated using du -sh) is 22GB and
> > the
> > > > > >>> broker crashes during startup, disk becomes full and no log
> more
> > is
> > > > > >>> written. (I can send an extraction of the logs, but the tell
> only
> > > > > about
> > > > > >>> 'rebuilding index', the same as on Java 8)
> > > > > >>>
> > > > > >>> Reverting the same broker to Java 8 and removing the index
> files,
> > > the
> > > > > >>> broker rebuilds such files, each files take 10MB, but the full
> > sum
> > > of
> > > > > sizes
> > > > > >>> (calculated using du -sh) is 38 MB !
> > > > > >>>
> > > > > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > > > > >>>
> > > > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0
> > (from
> > > > > >>> 0.10.2) without any success.
> > > > > >>>
> > > > > >>> All of the other testing clusters on CentOS7 (same SO settings)
> > did
> > > > not
> > > > > >>> have any problem.
> > > > > >>>
> > > > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size
> > (on
> > > > > >>> Java8 it works fine with 1GB and default max direct memory
> size)
> > > > > >>>
> > > > > >>> Which are the relevant configuration options ?
> > > > > >>>
> > > > > >>> Thoughts ?
> > > > > >>>
> > > > > >>> Thank you
> > > > > >>>
> > > > > >>> Enrico
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > --
> > >
> > >
> > > -- Enrico Olivelli
> > >
> >
> --
>
>
> -- Enrico Olivelli
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
Il sab 21 apr 2018, 06:29 Ismael Juma <is...@juma.me.uk> ha scritto:
> Hi Enrico,
>
> It is a real problem because it causes indexes to take a lot more disk
> space upfront. The sparsity is an important if people over partition, for
> example.
>
Got it.
In production I saw no issue, maybe due to much availability of disk space.
Should I file a JIRA or you will do?
Enrico
> Ismael
>
> On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha scritto:
> >
> > > Hi Enrico,
> > >
> > > Coincidentally, I saw your message to nio-dev and followed up there.
> > >
> >
> > I think this is not a 'real' problem, you will notice the difference only
> > if you have a lot of empty topics/partitions.
> > In fact when you simply upgrade to jdk9/10 immediately you are charged
> with
> > 10MB of disk space for each partition.
> > In production I did not suffer this change because usually you do not
> have
> > empty partitions.
> > In my test environments, where I had thousands of test empty partitions,
> > disks filled up immediately and without any reason, it took time to
> > understand the real cause. Broker will crash without much 'log' as disk
> is
> > out of space.
> >
> > Maybe it would be useful to add some notice about this potential problem
> > during the upgrade of the jdk.
> >
> > Hope that helps
> > Enrico
> >
> >
> > > Ismael
> > >
> > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > It is a deliberate change in JDK code
> > > >
> > > > Just for reference see this discussion on nio-dev list on OpenJDK
> > > >
> http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
> > > >
> > > >
> > > > see
> > > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > > >
> > > > Cheers
> > > > Enrico
> > > >
> > > >
> > > >
> > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > >
> > > > > Workaround:
> > > > > as these brokers are only for test environments I have set very
> small
> > > > > values for index file size, which affects pre-allocation
> > > > > segment.index.bytes=65536
> > > > > log.index.size.max.bytes=65536
> > > > >
> > > > > If anyone has some thought it will be very appreciated
> > > > > Cheers
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > > >
> > > > >> The only fact I have found is that with Java8 Kafka is creating
> > > "SPARSE"
> > > > >> files and with Java9 this is not true anymore
> > > > >>
> > > > >> Enrico
> > > > >>
> > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > > >>
> > > > >>> Hi,
> > > > >>> This is a very strage case. I have a Kafka broker (part of a
> > cluster
> > > of
> > > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to
> > > > Oracle JDK
> > > > >>> 9.0.4.
> > > > >>>
> > > > >>> There are a lot of .index and .timeindex files taking 10MB, they
> > are
> > > > for
> > > > >>> empty partiions.
> > > > >>>
> > > > >>> Running with Java 9 the server seems to rebuild these files and
> > each
> > > > >>> file takes "really" 10MB.
> > > > >>> The sum of all the files (calculated using du -sh) is 22GB and
> the
> > > > >>> broker crashes during startup, disk becomes full and no log more
> is
> > > > >>> written. (I can send an extraction of the logs, but the tell only
> > > > about
> > > > >>> 'rebuilding index', the same as on Java 8)
> > > > >>>
> > > > >>> Reverting the same broker to Java 8 and removing the index files,
> > the
> > > > >>> broker rebuilds such files, each files take 10MB, but the full
> sum
> > of
> > > > sizes
> > > > >>> (calculated using du -sh) is 38 MB !
> > > > >>>
> > > > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > > > >>>
> > > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0
> (from
> > > > >>> 0.10.2) without any success.
> > > > >>>
> > > > >>> All of the other testing clusters on CentOS7 (same SO settings)
> did
> > > not
> > > > >>> have any problem.
> > > > >>>
> > > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size
> (on
> > > > >>> Java8 it works fine with 1GB and default max direct memory size)
> > > > >>>
> > > > >>> Which are the relevant configuration options ?
> > > > >>>
> > > > >>> Thoughts ?
> > > > >>>
> > > > >>> Thank you
> > > > >>>
> > > > >>> Enrico
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
--
-- Enrico Olivelli
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Enrico,
It is a real problem because it causes indexes to take a lot more disk
space upfront. The sparsity is an important if people over partition, for
example.
Ismael
On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha scritto:
>
> > Hi Enrico,
> >
> > Coincidentally, I saw your message to nio-dev and followed up there.
> >
>
> I think this is not a 'real' problem, you will notice the difference only
> if you have a lot of empty topics/partitions.
> In fact when you simply upgrade to jdk9/10 immediately you are charged with
> 10MB of disk space for each partition.
> In production I did not suffer this change because usually you do not have
> empty partitions.
> In my test environments, where I had thousands of test empty partitions,
> disks filled up immediately and without any reason, it took time to
> understand the real cause. Broker will crash without much 'log' as disk is
> out of space.
>
> Maybe it would be useful to add some notice about this potential problem
> during the upgrade of the jdk.
>
> Hope that helps
> Enrico
>
>
> > Ismael
> >
> > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > It is a deliberate change in JDK code
> > >
> > > Just for reference see this discussion on nio-dev list on OpenJDK
> > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
> > >
> > >
> > > see
> > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > >
> > > Cheers
> > > Enrico
> > >
> > >
> > >
> > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > >
> > > > Workaround:
> > > > as these brokers are only for test environments I have set very small
> > > > values for index file size, which affects pre-allocation
> > > > segment.index.bytes=65536
> > > > log.index.size.max.bytes=65536
> > > >
> > > > If anyone has some thought it will be very appreciated
> > > > Cheers
> > > >
> > > > Enrico
> > > >
> > > >
> > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > >
> > > >> The only fact I have found is that with Java8 Kafka is creating
> > "SPARSE"
> > > >> files and with Java9 this is not true anymore
> > > >>
> > > >> Enrico
> > > >>
> > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > > >>
> > > >>> Hi,
> > > >>> This is a very strage case. I have a Kafka broker (part of a
> cluster
> > of
> > > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to
> > > Oracle JDK
> > > >>> 9.0.4.
> > > >>>
> > > >>> There are a lot of .index and .timeindex files taking 10MB, they
> are
> > > for
> > > >>> empty partiions.
> > > >>>
> > > >>> Running with Java 9 the server seems to rebuild these files and
> each
> > > >>> file takes "really" 10MB.
> > > >>> The sum of all the files (calculated using du -sh) is 22GB and the
> > > >>> broker crashes during startup, disk becomes full and no log more is
> > > >>> written. (I can send an extraction of the logs, but the tell only
> > > about
> > > >>> 'rebuilding index', the same as on Java 8)
> > > >>>
> > > >>> Reverting the same broker to Java 8 and removing the index files,
> the
> > > >>> broker rebuilds such files, each files take 10MB, but the full sum
> of
> > > sizes
> > > >>> (calculated using du -sh) is 38 MB !
> > > >>>
> > > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > > >>>
> > > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> > > >>> 0.10.2) without any success.
> > > >>>
> > > >>> All of the other testing clusters on CentOS7 (same SO settings) did
> > not
> > > >>> have any problem.
> > > >>>
> > > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on
> > > >>> Java8 it works fine with 1GB and default max direct memory size)
> > > >>>
> > > >>> Which are the relevant configuration options ?
> > > >>>
> > > >>> Thoughts ?
> > > >>>
> > > >>> Thank you
> > > >>>
> > > >>> Enrico
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > >
> >
> --
>
>
> -- Enrico Olivelli
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
Il ven 20 apr 2018, 20:24 Ismael Juma <is...@juma.me.uk> ha scritto:
> Hi Enrico,
>
> Coincidentally, I saw your message to nio-dev and followed up there.
>
I think this is not a 'real' problem, you will notice the difference only
if you have a lot of empty topics/partitions.
In fact when you simply upgrade to jdk9/10 immediately you are charged with
10MB of disk space for each partition.
In production I did not suffer this change because usually you do not have
empty partitions.
In my test environments, where I had thousands of test empty partitions,
disks filled up immediately and without any reason, it took time to
understand the real cause. Broker will crash without much 'log' as disk is
out of space.
Maybe it would be useful to add some notice about this potential problem
during the upgrade of the jdk.
Hope that helps
Enrico
> Ismael
>
> On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > It is a deliberate change in JDK code
> >
> > Just for reference see this discussion on nio-dev list on OpenJDK
> > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
> >
> >
> > see
> > https://bugs.openjdk.java.net/browse/JDK-8168628
> >
> > Cheers
> > Enrico
> >
> >
> >
> > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> >
> > > Workaround:
> > > as these brokers are only for test environments I have set very small
> > > values for index file size, which affects pre-allocation
> > > segment.index.bytes=65536
> > > log.index.size.max.bytes=65536
> > >
> > > If anyone has some thought it will be very appreciated
> > > Cheers
> > >
> > > Enrico
> > >
> > >
> > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > >
> > >> The only fact I have found is that with Java8 Kafka is creating
> "SPARSE"
> > >> files and with Java9 this is not true anymore
> > >>
> > >> Enrico
> > >>
> > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> > >>
> > >>> Hi,
> > >>> This is a very strage case. I have a Kafka broker (part of a cluster
> of
> > >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to
> > Oracle JDK
> > >>> 9.0.4.
> > >>>
> > >>> There are a lot of .index and .timeindex files taking 10MB, they are
> > for
> > >>> empty partiions.
> > >>>
> > >>> Running with Java 9 the server seems to rebuild these files and each
> > >>> file takes "really" 10MB.
> > >>> The sum of all the files (calculated using du -sh) is 22GB and the
> > >>> broker crashes during startup, disk becomes full and no log more is
> > >>> written. (I can send an extraction of the logs, but the tell only
> > about
> > >>> 'rebuilding index', the same as on Java 8)
> > >>>
> > >>> Reverting the same broker to Java 8 and removing the index files, the
> > >>> broker rebuilds such files, each files take 10MB, but the full sum of
> > sizes
> > >>> (calculated using du -sh) is 38 MB !
> > >>>
> > >>> I am running this broker on CentosOS 7 on EXT4 FS.
> > >>>
> > >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> > >>> 0.10.2) without any success.
> > >>>
> > >>> All of the other testing clusters on CentOS7 (same SO settings) did
> not
> > >>> have any problem.
> > >>>
> > >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on
> > >>> Java8 it works fine with 1GB and default max direct memory size)
> > >>>
> > >>> Which are the relevant configuration options ?
> > >>>
> > >>> Thoughts ?
> > >>>
> > >>> Thank you
> > >>>
> > >>> Enrico
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >
> >
>
--
-- Enrico Olivelli
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Enrico,
Coincidentally, I saw your message to nio-dev and followed up there.
Ismael
On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> It is a deliberate change in JDK code
>
> Just for reference see this discussion on nio-dev list on OpenJDK
> http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
>
>
> see
> https://bugs.openjdk.java.net/browse/JDK-8168628
>
> Cheers
> Enrico
>
>
>
> 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
>
> > Workaround:
> > as these brokers are only for test environments I have set very small
> > values for index file size, which affects pre-allocation
> > segment.index.bytes=65536
> > log.index.size.max.bytes=65536
> >
> > If anyone has some thought it will be very appreciated
> > Cheers
> >
> > Enrico
> >
> >
> > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> >
> >> The only fact I have found is that with Java8 Kafka is creating "SPARSE"
> >> files and with Java9 this is not true anymore
> >>
> >> Enrico
> >>
> >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> >>
> >>> Hi,
> >>> This is a very strage case. I have a Kafka broker (part of a cluster of
> >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to
> Oracle JDK
> >>> 9.0.4.
> >>>
> >>> There are a lot of .index and .timeindex files taking 10MB, they are
> for
> >>> empty partiions.
> >>>
> >>> Running with Java 9 the server seems to rebuild these files and each
> >>> file takes "really" 10MB.
> >>> The sum of all the files (calculated using du -sh) is 22GB and the
> >>> broker crashes during startup, disk becomes full and no log more is
> >>> written. (I can send an extraction of the logs, but the tell only
> about
> >>> 'rebuilding index', the same as on Java 8)
> >>>
> >>> Reverting the same broker to Java 8 and removing the index files, the
> >>> broker rebuilds such files, each files take 10MB, but the full sum of
> sizes
> >>> (calculated using du -sh) is 38 MB !
> >>>
> >>> I am running this broker on CentosOS 7 on EXT4 FS.
> >>>
> >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> >>> 0.10.2) without any success.
> >>>
> >>> All of the other testing clusters on CentOS7 (same SO settings) did not
> >>> have any problem.
> >>>
> >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on
> >>> Java8 it works fine with 1GB and default max direct memory size)
> >>>
> >>> Which are the relevant configuration options ?
> >>>
> >>> Thoughts ?
> >>>
> >>> Thank you
> >>>
> >>> Enrico
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
It is a deliberate change in JDK code
Just for reference see this discussion on nio-dev list on OpenJDK
http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
see
https://bugs.openjdk.java.net/browse/JDK-8168628
Cheers
Enrico
2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> Workaround:
> as these brokers are only for test environments I have set very small
> values for index file size, which affects pre-allocation
> segment.index.bytes=65536
> log.index.size.max.bytes=65536
>
> If anyone has some thought it will be very appreciated
> Cheers
>
> Enrico
>
>
> 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
>
>> The only fact I have found is that with Java8 Kafka is creating "SPARSE"
>> files and with Java9 this is not true anymore
>>
>> Enrico
>>
>> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
>>
>>> Hi,
>>> This is a very strage case. I have a Kafka broker (part of a cluster of
>>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
>>> 9.0.4.
>>>
>>> There are a lot of .index and .timeindex files taking 10MB, they are for
>>> empty partiions.
>>>
>>> Running with Java 9 the server seems to rebuild these files and each
>>> file takes "really" 10MB.
>>> The sum of all the files (calculated using du -sh) is 22GB and the
>>> broker crashes during startup, disk becomes full and no log more is
>>> written. (I can send an extraction of the logs, but the tell only about
>>> 'rebuilding index', the same as on Java 8)
>>>
>>> Reverting the same broker to Java 8 and removing the index files, the
>>> broker rebuilds such files, each files take 10MB, but the full sum of sizes
>>> (calculated using du -sh) is 38 MB !
>>>
>>> I am running this broker on CentosOS 7 on EXT4 FS.
>>>
>>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
>>> 0.10.2) without any success.
>>>
>>> All of the other testing clusters on CentOS7 (same SO settings) did not
>>> have any problem.
>>>
>>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on
>>> Java8 it works fine with 1GB and default max direct memory size)
>>>
>>> Which are the relevant configuration options ?
>>>
>>> Thoughts ?
>>>
>>> Thank you
>>>
>>> Enrico
>>>
>>>
>>>
>>>
>>
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
Workaround:
as these brokers are only for test environments I have set very small
values for index file size, which affects pre-allocation
segment.index.bytes=65536
log.index.size.max.bytes=65536
If anyone has some thought it will be very appreciated
Cheers
Enrico
2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> The only fact I have found is that with Java8 Kafka is creating "SPARSE"
> files and with Java9 this is not true anymore
>
> Enrico
>
> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
>
>> Hi,
>> This is a very strage case. I have a Kafka broker (part of a cluster of 3
>> brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
>> 9.0.4.
>>
>> There are a lot of .index and .timeindex files taking 10MB, they are for
>> empty partiions.
>>
>> Running with Java 9 the server seems to rebuild these files and each file
>> takes "really" 10MB.
>> The sum of all the files (calculated using du -sh) is 22GB and the broker
>> crashes during startup, disk becomes full and no log more is written. (I
>> can send an extraction of the logs, but the tell only about 'rebuilding
>> index', the same as on Java 8)
>>
>> Reverting the same broker to Java 8 and removing the index files, the
>> broker rebuilds such files, each files take 10MB, but the full sum of sizes
>> (calculated using du -sh) is 38 MB !
>>
>> I am running this broker on CentosOS 7 on EXT4 FS.
>>
>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
>> 0.10.2) without any success.
>>
>> All of the other testing clusters on CentOS7 (same SO settings) did not
>> have any problem.
>>
>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8
>> it works fine with 1GB and default max direct memory size)
>>
>> Which are the relevant configuration options ?
>>
>> Thoughts ?
>>
>> Thank you
>>
>> Enrico
>>
>>
>>
>>
>
Re: Broker cannot start switch to Java9 - weird file system issue ?
Posted by Enrico Olivelli <eo...@gmail.com>.
The only fact I have found is that with Java8 Kafka is creating "SPARSE"
files and with Java9 this is not true anymore
Enrico
2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> Hi,
> This is a very strage case. I have a Kafka broker (part of a cluster of 3
> brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
> 9.0.4.
>
> There are a lot of .index and .timeindex files taking 10MB, they are for
> empty partiions.
>
> Running with Java 9 the server seems to rebuild these files and each file
> takes "really" 10MB.
> The sum of all the files (calculated using du -sh) is 22GB and the broker
> crashes during startup, disk becomes full and no log more is written. (I
> can send an extraction of the logs, but the tell only about 'rebuilding
> index', the same as on Java 8)
>
> Reverting the same broker to Java 8 and removing the index files, the
> broker rebuilds such files, each files take 10MB, but the full sum of sizes
> (calculated using du -sh) is 38 MB !
>
> I am running this broker on CentosOS 7 on EXT4 FS.
>
> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> 0.10.2) without any success.
>
> All of the other testing clusters on CentOS7 (same SO settings) did not
> have any problem.
>
> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8
> it works fine with 1GB and default max direct memory size)
>
> Which are the relevant configuration options ?
>
> Thoughts ?
>
> Thank you
>
> Enrico
>
>
>
>