You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ilya R <mr...@gmail.com> on 2020/04/16 15:28:01 UTC
Kafka slow reads from beginning of topic
Hello, everyone
I'm researching problem when reading a topic from the beginning max speed per partition/per consumer not more than 80MB/s. But near the end of the topic speed rise up to 250-300MB/s unexpectedly, not only on the last log segment on several latest log segments. I've already change network buffers in a linux, change socket settings on server/client side, but there are not results. For testing I'm using kafka-console-consimer.sh script from standart installation. Kafka version is 2.3.0.
Does anyone else encountered this?
Server settings
broker.id=1
log.message.timestamp.type=LogAppendTime
num.network.threads=24
num.io.threads=10
offset.metadata.max.bytes=8096
offsets.load.buffer.size=10485760
offsets.topic.num.partitions=1
offsets.topic.replication.factor=1
socket.request.max.bytes=4194304
socket.send.buffer.bytes=4194304
leader.imbalance.check.interval.seconds = 30
log.dir=/local/kafka/data
log.dirs=/local/kafka/data
zookeeper.connect=localhost:2181
message.max.bytes = 50000000
log.segment.bytes = 104857600
Topic settings:
compression.type uncompressed
segment.bytes 104857600
max.message.bytes 50000000
min.compaction.lag.ms 43200000
message.timestamp.type LogAppendTime
min.cleanable.dirty.ratio 0.1
Consumer settings:
max.partition.fetch.bytes=8048576
fetch.max.bytes=128777216
fetch.min.bytes=8048576
receive.buffer.bytes=8048577
Re: Kafka slow reads from beginning of topic
Posted by Ilya R <mr...@gmail.com>.
There is RAID10 on 10 SSD Intel 1.9TB with raid cache 2GB. Topic size is 290GB it nearly fully in cache. Broker CPU usage near 2-3%.
CPU is 2 x Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz.
I didn't find any bottlenecks via JMX graphs.
On 2020/04/16 15:59:23, Seva Feldman <se...@ironsrc.com> wrote:
> Hi,
>
> Can you look at the disk IO while you are approaching end of the topic and
> your performance raises? End of the topic may reside in FS cache in memory
> and that may explain.
>
> BR
>
> On Thu, Apr 16, 2020 at 6:56 PM Ilya R <mr...@gmail.com> wrote:
>
> > Hello, everyone
> >
> > I'm researching problem when reading a topic from the beginning max speed
> > per partition/per consumer not more than 80MB/s. But near the end of the
> > topic speed rise up to 250-300MB/s unexpectedly, not only on the last log
> > segment on several latest log segments. I've already change network buffers
> > in a linux, change socket settings on server/client side, but there are not
> > results. For testing I'm using kafka-console-consimer.sh script from
> > standart installation. Kafka version is 2.3.0.
> >
> > Does anyone else encountered this?
> >
> > Server settings
> > broker.id=1
> > log.message.timestamp.type=LogAppendTime
> > num.network.threads=24
> > num.io.threads=10
> > offset.metadata.max.bytes=8096
> > offsets.load.buffer.size=10485760
> > offsets.topic.num.partitions=1
> > offsets.topic.replication.factor=1
> > socket.request.max.bytes=4194304
> > socket.send.buffer.bytes=4194304
> > leader.imbalance.check.interval.seconds = 30
> > log.dir=/local/kafka/data
> > log.dirs=/local/kafka/data
> > zookeeper.connect=localhost:2181
> > message.max.bytes = 50000000
> > log.segment.bytes = 104857600
> >
> > Topic settings:
> > compression.type uncompressed
> > segment.bytes 104857600
> > max.message.bytes 50000000
> > min.compaction.lag.ms 43200000
> > message.timestamp.type LogAppendTime
> > min.cleanable.dirty.ratio 0.1
> >
> > Consumer settings:
> > max.partition.fetch.bytes=8048576
> > fetch.max.bytes=128777216
> > fetch.min.bytes=8048576
> > receive.buffer.bytes=8048577
> >
>
>
> --
> Seva Feldman
> VP R&D Mobile Delivery
> [image: ironSource] <http://www.ironsrc.com/>
>
> email seva.f@ironsrc.com
> mobile +972544346089
>
> ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
>
Re: Kafka slow reads from beginning of topic
Posted by Seva Feldman <se...@ironsrc.com>.
Hi,
Can you look at the disk IO while you are approaching end of the topic and
your performance raises? End of the topic may reside in FS cache in memory
and that may explain.
BR
On Thu, Apr 16, 2020 at 6:56 PM Ilya R <mr...@gmail.com> wrote:
> Hello, everyone
>
> I'm researching problem when reading a topic from the beginning max speed
> per partition/per consumer not more than 80MB/s. But near the end of the
> topic speed rise up to 250-300MB/s unexpectedly, not only on the last log
> segment on several latest log segments. I've already change network buffers
> in a linux, change socket settings on server/client side, but there are not
> results. For testing I'm using kafka-console-consimer.sh script from
> standart installation. Kafka version is 2.3.0.
>
> Does anyone else encountered this?
>
> Server settings
> broker.id=1
> log.message.timestamp.type=LogAppendTime
> num.network.threads=24
> num.io.threads=10
> offset.metadata.max.bytes=8096
> offsets.load.buffer.size=10485760
> offsets.topic.num.partitions=1
> offsets.topic.replication.factor=1
> socket.request.max.bytes=4194304
> socket.send.buffer.bytes=4194304
> leader.imbalance.check.interval.seconds = 30
> log.dir=/local/kafka/data
> log.dirs=/local/kafka/data
> zookeeper.connect=localhost:2181
> message.max.bytes = 50000000
> log.segment.bytes = 104857600
>
> Topic settings:
> compression.type uncompressed
> segment.bytes 104857600
> max.message.bytes 50000000
> min.compaction.lag.ms 43200000
> message.timestamp.type LogAppendTime
> min.cleanable.dirty.ratio 0.1
>
> Consumer settings:
> max.partition.fetch.bytes=8048576
> fetch.max.bytes=128777216
> fetch.min.bytes=8048576
> receive.buffer.bytes=8048577
>
--
Seva Feldman
VP R&D Mobile Delivery
[image: ironSource] <http://www.ironsrc.com/>
email seva.f@ironsrc.com
mobile +972544346089
ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv