You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vishnu Manivannan <vi...@arrcus.com> on 2018/07/18 19:00:50 UTC
Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu
RE: Kafka Connect: Increase Consumer Consumption
Posted by adrien ruffie <ad...@hotmail.fr>.
At this point of investigation, I have no idea....
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : vendredi 20 juillet 2018 21:28:24
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
No, I'm just running one consumer now and trying to increase its consumption before I increase the number of consumers.
Yeah I know it's weird, I thought I would see some changes in data consumption after tweaking these parameters.
On 7/18/18, 11:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Strange enough ...
I don't really understand why.
Do you have multiple consumer ?
I don't know if a metric which gives the external consumption of a topic.
But if even simple metric exist, you can check if it's your consumer which not really receive more than 1150 records,
and your topic gets out more than 5000 records.
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : jeudi 19 juillet 2018 00:05:14
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
Hi Adrien,
I set fetch.max.wait.ms to 1500 ms and ran it again. It still isn't crossing 1150 records per fetch. On the producer side (using kakfa-producer-perf-test), its producing about 30,000 records/sec and a million records in total.
I tried different configurations amongst these three parameters (max.poll.records, fetch.min.bytes, fetch.max.wait.ms). All the other parameters are unchanged and have their default values. I have not changed any of the broker configs either. I read the Kafka documentation but could not find any other parameters that could impact the fetch size.
Is there some other consumer or broker parameter that I might be missing?
Thanks,
Vishnu
On 7/18/18, 1:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000 records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer. fetch.max.wait.ms lets you control how long to wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra latency in case there is not enough data flowing to the Kafka topic to satisfy the minimum amount of data to return. If you want to limit the potential latency (usually due to SLAs controlling the maximum latency of the application), you can set fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu
Re: Kafka Connect: Increase Consumer Consumption
Posted by Vishnu Manivannan <vi...@arrcus.com>.
No, I'm just running one consumer now and trying to increase its consumption before I increase the number of consumers.
Yeah I know it's weird, I thought I would see some changes in data consumption after tweaking these parameters.
On 7/18/18, 11:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Strange enough ...
I don't really understand why.
Do you have multiple consumer ?
I don't know if a metric which gives the external consumption of a topic.
But if even simple metric exist, you can check if it's your consumer which not really receive more than 1150 records,
and your topic gets out more than 5000 records.
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : jeudi 19 juillet 2018 00:05:14
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
Hi Adrien,
I set fetch.max.wait.ms to 1500 ms and ran it again. It still isn't crossing 1150 records per fetch. On the producer side (using kakfa-producer-perf-test), its producing about 30,000 records/sec and a million records in total.
I tried different configurations amongst these three parameters (max.poll.records, fetch.min.bytes, fetch.max.wait.ms). All the other parameters are unchanged and have their default values. I have not changed any of the broker configs either. I read the Kafka documentation but could not find any other parameters that could impact the fetch size.
Is there some other consumer or broker parameter that I might be missing?
Thanks,
Vishnu
On 7/18/18, 1:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000 records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer. fetch.max.wait.ms lets you control how long to wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra latency in case there is not enough data flowing to the Kafka topic to satisfy the minimum amount of data to return. If you want to limit the potential latency (usually due to SLAs controlling the maximum latency of the application), you can set fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu
RE: Kafka Connect: Increase Consumer Consumption
Posted by adrien ruffie <ad...@hotmail.fr>.
Strange enough ...
I don't really understand why.
Do you have multiple consumer ?
I don't know if a metric which gives the external consumption of a topic.
But if even simple metric exist, you can check if it's your consumer which not really receive more than 1150 records,
and your topic gets out more than 5000 records.
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : jeudi 19 juillet 2018 00:05:14
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
Hi Adrien,
I set fetch.max.wait.ms to 1500 ms and ran it again. It still isn't crossing 1150 records per fetch. On the producer side (using kakfa-producer-perf-test), its producing about 30,000 records/sec and a million records in total.
I tried different configurations amongst these three parameters (max.poll.records, fetch.min.bytes, fetch.max.wait.ms). All the other parameters are unchanged and have their default values. I have not changed any of the broker configs either. I read the Kafka documentation but could not find any other parameters that could impact the fetch size.
Is there some other consumer or broker parameter that I might be missing?
Thanks,
Vishnu
On 7/18/18, 1:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000 records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer. fetch.max.wait.ms lets you control how long to wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra latency in case there is not enough data flowing to the Kafka topic to satisfy the minimum amount of data to return. If you want to limit the potential latency (usually due to SLAs controlling the maximum latency of the application), you can set fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu
Re: Kafka Connect: Increase Consumer Consumption
Posted by Vishnu Manivannan <vi...@arrcus.com>.
Hi Adrien,
I set fetch.max.wait.ms to 1500 ms and ran it again. It still isn't crossing 1150 records per fetch. On the producer side (using kakfa-producer-perf-test), its producing about 30,000 records/sec and a million records in total.
I tried different configurations amongst these three parameters (max.poll.records, fetch.min.bytes, fetch.max.wait.ms). All the other parameters are unchanged and have their default values. I have not changed any of the broker configs either. I read the Kafka documentation but could not find any other parameters that could impact the fetch size.
Is there some other consumer or broker parameter that I might be missing?
Thanks,
Vishnu
On 7/18/18, 1:48 PM, "adrien ruffie" <ad...@hotmail.fr> wrote:
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000 records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer. fetch.max.wait.ms lets you control how long to wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra latency in case there is not enough data flowing to the Kafka topic to satisfy the minimum amount of data to return. If you want to limit the potential latency (usually due to SLAs controlling the maximum latency of the application), you can set fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu
RE: Kafka Connect: Increase Consumer Consumption
Posted by adrien ruffie <ad...@hotmail.fr>.
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000 records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer. fetch.max.wait.ms lets you control how long to wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra latency in case there is not enough data flowing to the Kafka topic to satisfy the minimum amount of data to return. If you want to limit the potential latency (usually due to SLAs controlling the maximum latency of the application), you can set fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vi...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka consumer. I am trying to get the consumer to fetch more records, so I can increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000 records each time. So, I changed the configurations for the following consumer parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does not go above 1150. Are there any other parameters that I should look into or any changes I should make to the current configurations?
Thanks,
Vishnu