You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jose Manuel Vega Monroy <jo...@williamhill.com> on 2019/09/02 10:18:38 UTC
Re: [EXTERNAL] Re: Kafka - Possible messages order
Hi there,
Finally we enabled idempotence, and setting as per official documentation max.in.flight.requests.per.connection <= 5 and acks = all.
'enable.idempotence' => 'true',
'max.in.flight.requests.per.connection' => '4',
'acks' => 'all',
" When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConfigException will be thrown."
And now facing problems to create producer, forcing max.in.flight.requests.per.connection = 1:
Caused by: org.apache.kafka.common.config.ConfigException: Must set max.in.flight.requests.per.connection to 1 in orderto use the idempotent producer. Otherwise we cannot guarantee idempotence.
at org.apache.kafka.clients.producer.KafkaProducer.configureInflightRequests(KafkaProducer.java:482)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:360)
... 17 common frames omitted
Any idea about?
Our Kafka broker version is 2.2.1.
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 29/08/2019, 17:47, "Anatoliy Soldatov" <ak...@avito.ru.INVALID> wrote:
What exactly is the problem?
Kafka states it in documentation and it is not new.
Quote from docs: max.in.flight.requests.per.connection – The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).
The possible problem is only with some people not able to read documentation.
29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy <jo...@williamhill.com>> написал(а):
Hi there,
Recently we found this blog entry about a possible problem with messages order: https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.softwaremill.com_does-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=eoZO2gqSx3WfGCIIcane_DH28EY2j1xHy5pwK9o70L4&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fblog.softwaremill.com-252Fdoes-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034629468-26sdata-3D-252BulzkEeXQ6xhczBYfYQ8wnvWrpW-252Fs3wVtQaGGwIgPbM-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=dFwxCXAt09CuEx1CL20C8BR4SKDsWzZgfdIQ3U6ZAi8&e= >
Please could you confirm about? And if so, how to fix? Blog entry suggesting how to.
Thanks
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ph2WXQKWT4O1uhip6cQzhhNHSKnxw6S5xChr13VQmMk&e= ]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.williamhill.com-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034639463-26sdata-3DKowmt6DYcYKQgMC-252FI5Yv-252BeZNsaPhh8oK8-252F9jpvfyuSk-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=T7tzoiDU2XkOMiQ_Hq0KEBKH3DoW7lM2Hv2jGqwutjc&e= >
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_senet.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=_o5y3RjRJ1q8K5sKGbyP0-Ryn19BzNYT9gG16G_BDO8&e= ]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.whenthefunstops.co.uk-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034649456-26sdata-3D-252FtXL-252FsYYwa2yGdPRNQ7-252FpYAos0CkWYxaoVqwLJP0Xfs-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ezMzWlYi-DwoK2lzhJC3AQltVduPff9GdL51lWI5wSk&e= >
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com<ma...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
________________________________
"This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”
Re: [EXTERNAL] Re: Kafka - Possible messages order
Posted by Jose Manuel Vega Monroy <jo...@williamhill.com>.
Hi there,
I need to verify, but probably due to our Kafka client being so old.
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 02/09/2019, 12:15, "Jose Manuel Vega Monroy" <jo...@williamhill.com> wrote:
Hi there,
Finally we enabled idempotence, and setting as per official documentation max.in.flight.requests.per.connection <= 5 and acks = all.
'enable.idempotence' => 'true',
'max.in.flight.requests.per.connection' => '4',
'acks' => 'all',
" When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConfigException will be thrown."
And now facing problems to create producer, forcing max.in.flight.requests.per.connection = 1:
Caused by: org.apache.kafka.common.config.ConfigException: Must set max.in.flight.requests.per.connection to 1 in orderto use the idempotent producer. Otherwise we cannot guarantee idempotence.
at org.apache.kafka.clients.producer.KafkaProducer.configureInflightRequests(KafkaProducer.java:482)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:360)
... 17 common frames omitted
Any idea about?
Our Kafka broker version is 2.2.1.
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 29/08/2019, 17:47, "Anatoliy Soldatov" <ak...@avito.ru.INVALID> wrote:
What exactly is the problem?
Kafka states it in documentation and it is not new.
Quote from docs: max.in.flight.requests.per.connection – The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).
The possible problem is only with some people not able to read documentation.
29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy <jo...@williamhill.com>> написал(а):
Hi there,
Recently we found this blog entry about a possible problem with messages order: https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.softwaremill.com_does-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=eoZO2gqSx3WfGCIIcane_DH28EY2j1xHy5pwK9o70L4&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fblog.softwaremill.com-252Fdoes-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034629468-26sdata-3D-252BulzkEeXQ6xhczBYfYQ8wnvWrpW-252Fs3wVtQaGGwIgPbM-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=dFwxCXAt09CuEx1CL20C8BR4SKDsWzZgfdIQ3U6ZAi8&e= >
Please could you confirm about? And if so, how to fix? Blog entry suggesting how to.
Thanks
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ph2WXQKWT4O1uhip6cQzhhNHSKnxw6S5xChr13VQmMk&e= ]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.williamhill.com-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034639463-26sdata-3DKowmt6DYcYKQgMC-252FI5Yv-252BeZNsaPhh8oK8-252F9jpvfyuSk-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=T7tzoiDU2XkOMiQ_Hq0KEBKH3DoW7lM2Hv2jGqwutjc&e= >
[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_senet.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=_o5y3RjRJ1q8K5sKGbyP0-Ryn19BzNYT9gG16G_BDO8&e= ]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.whenthefunstops.co.uk-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034649456-26sdata-3D-252FtXL-252FsYYwa2yGdPRNQ7-252FpYAos0CkWYxaoVqwLJP0Xfs-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ezMzWlYi-DwoK2lzhJC3AQltVduPff9GdL51lWI5wSk&e= >
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com<ma...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
________________________________
"This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may not copy, save, print or forward it to any third party and you are kindly requested to destroy this message and notify the sender thereof by email.
Данное сообщение содержит конфиденциальную информацию/информацию, являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом данного сообщения, Вы не вправе копировать, сохранять, печатать или пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и уведомить об этом отправителя электронным письмом.”