You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ilya R <mr...@gmail.com> on 2020/04/16 15:49:16 UTC
Kafka slow reads from beggining of topic
Hello, everyone
I'm researching problem when reading a topic from the beginning max speed per partition/per consumer not more than 80MB/s. But near the end of the topic speed rise up to 250-300MB/s unexpectedly, not only on the last log segment on several latest log segments. I've already change network buffers in a linux, change socket settings on server/client side, but there are not results. For testing I'm using kafka-console-consimer.sh script from standart installation. Kafka version is 2.3.0.
Does anyone else encountered this?
Server settings
broker.id=1
log.message.timestamp.type=LogAppendTime
num.network.threads=24
num.io.threads=10
offset.metadata.max.bytes=8096
offsets.load.buffer.size=10485760
offsets.topic.num.partitions=1
offsets.topic.replication.factor=1
socket.request.max.bytes=4194304
socket.send.buffer.bytes=4194304
leader.imbalance.check.interval.seconds = 30
log.dir=/local/kafka/data
log.dirs=/local/kafka/data
zookeeper.connect=localhost:2181
message.max.bytes = 50000000
log.segment.bytes = 104857600
Topic settings:
compression.type uncompressed
segment.bytes 104857600
max.message.bytes 50000000
min.compaction.lag.ms 43200000
message.timestamp.type LogAppendTime
min.cleanable.dirty.ratio 0.1
Consumer settings:
max.partition.fetch.bytes=8048576
fetch.max.bytes=128777216
fetch.min.bytes=8048576
receive.buffer.bytes=8048577
Re: Kafka slow reads from beggining of topic
Posted by Ilya R <mr...@gmail.com>.
Sorry sent several times, ignore please this thread.
On 2020/04/16 15:49:16, Ilya R <mr...@gmail.com> wrote:
> Hello, everyone
>
> I'm researching problem when reading a topic from the beginning max speed per partition/per consumer not more than 80MB/s. But near the end of the topic speed rise up to 250-300MB/s unexpectedly, not only on the last log segment on several latest log segments. I've already change network buffers in a linux, change socket settings on server/client side, but there are not results. For testing I'm using kafka-console-consimer.sh script from standart installation. Kafka version is 2.3.0.
>
> Does anyone else encountered this?
>
> Server settings
> broker.id=1
> log.message.timestamp.type=LogAppendTime
> num.network.threads=24
> num.io.threads=10
> offset.metadata.max.bytes=8096
> offsets.load.buffer.size=10485760
> offsets.topic.num.partitions=1
> offsets.topic.replication.factor=1
> socket.request.max.bytes=4194304
> socket.send.buffer.bytes=4194304
> leader.imbalance.check.interval.seconds = 30
> log.dir=/local/kafka/data
> log.dirs=/local/kafka/data
> zookeeper.connect=localhost:2181
> message.max.bytes = 50000000
> log.segment.bytes = 104857600
>
> Topic settings:
> compression.type uncompressed
> segment.bytes 104857600
> max.message.bytes 50000000
> min.compaction.lag.ms 43200000
> message.timestamp.type LogAppendTime
> min.cleanable.dirty.ratio 0.1
>
> Consumer settings:
> max.partition.fetch.bytes=8048576
> fetch.max.bytes=128777216
> fetch.min.bytes=8048576
> receive.buffer.bytes=8048577
>