You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "JEVTIC, MARKO" <MA...@fisglobal.com> on 2017/05/30 13:04:34 UTC
client recordmetadata meaning
Hi all,
I wasn't able to find in documentation firm agreement about Kafka message reply.
So, before going through the source code, I would like to ask a question:
If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure, provided disks didn't crash ?
Thank you for your time.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: client recordmetadata meaning
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
I'm not sure if the flush would happen before the ack. Maybe somebody closer to the code can answer that? I haven't tested but I think your performance will go way down.
-Dave
-----Original Message-----
From: JEVTIC, MARKO [mailto:MARKO.JEVTIC@fisglobal.com]
Sent: Tuesday, May 30, 2017 10:49 AM
To: users@kafka.apache.org
Subject: Re: client recordmetadata meaning
Thank you Dave.
But if we have configuration of:
log.flush.interval.messages=1
Could we assume that without replication, when we get the client producer reply, we assume that message has been written to the disk.
It's somewhat not clear to me in the documentation, whether log.flush.interval.messages is connected to the ACK sent to the client.
________________________________
From: Tauzell, Dave <Da...@surescripts.com>
Sent: Tuesday, May 30, 2017 5:08:04 PM
To: users@kafka.apache.org
Subject: RE: client recordmetadata meaning
>>If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
No, it doesn't. The meaning depends on your configuration (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com%2Fdocumentation%2Fkafka%2Flatest%2Ftopics%2Fkafka_ha.html&data=01%7C01%7CMARKO.JEVTIC%40fisglobal.com%7Cfc2f91295c12427b395c08d4a76db963%7Ce3ff91d834c84b15a0b418910a6ac575%7C0&sdata=Tn%2BZIwosrxKC4cMEzwc01of7JTzwSLNynd9GYt1x0kw%3D&reserved=0). To increase the durability of messages you can have an acknowledgement mean that the broker has received the messages and replicated it to N other brokers (although they may not have actually fsynced to disk either).
>> My next related question is, if we have a single kafka server/broker,
>>could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure, provided disks didn't crash ?
No, see my above reply.
-Dave
-----Original Message-----
From: JEVTIC, MARKO [mailto:MARKO.JEVTIC@fisglobal.com]
Sent: Tuesday, May 30, 2017 8:05 AM
To: users@kafka.apache.org
Subject: client recordmetadata meaning
Hi all,
I wasn't able to find in documentation firm agreement about Kafka message reply.
So, before going through the source code, I would like to ask a question:
If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure, provided disks didn't crash ?
Thank you for your time.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: client recordmetadata meaning
Posted by "JEVTIC, MARKO" <MA...@fisglobal.com>.
Thank you Dave.
But if we have configuration of:
log.flush.interval.messages=1
Could we assume that without replication, when we get the client producer reply, we assume that message has been written to the disk.
It's somewhat not clear to me in the documentation, whether log.flush.interval.messages is connected to the ACK sent to the client.
________________________________
From: Tauzell, Dave <Da...@surescripts.com>
Sent: Tuesday, May 30, 2017 5:08:04 PM
To: users@kafka.apache.org
Subject: RE: client recordmetadata meaning
>>If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
No, it doesn't. The meaning depends on your configuration (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com%2Fdocumentation%2Fkafka%2Flatest%2Ftopics%2Fkafka_ha.html&data=01%7C01%7CMARKO.JEVTIC%40fisglobal.com%7Cfc2f91295c12427b395c08d4a76db963%7Ce3ff91d834c84b15a0b418910a6ac575%7C0&sdata=Tn%2BZIwosrxKC4cMEzwc01of7JTzwSLNynd9GYt1x0kw%3D&reserved=0). To increase the durability of messages you can have an acknowledgement mean that the broker has received the messages and replicated it to N other brokers (although they may not have actually fsynced to disk either).
>> My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure,
>>provided disks didn't crash ?
No, see my above reply.
-Dave
-----Original Message-----
From: JEVTIC, MARKO [mailto:MARKO.JEVTIC@fisglobal.com]
Sent: Tuesday, May 30, 2017 8:05 AM
To: users@kafka.apache.org
Subject: client recordmetadata meaning
Hi all,
I wasn't able to find in documentation firm agreement about Kafka message reply.
So, before going through the source code, I would like to ask a question:
If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure, provided disks didn't crash ?
Thank you for your time.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: client recordmetadata meaning
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
>>If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
No, it doesn't. The meaning depends on your configuration (https://www.cloudera.com/documentation/kafka/latest/topics/kafka_ha.html). To increase the durability of messages you can have an acknowledgement mean that the broker has received the messages and replicated it to N other brokers (although they may not have actually fsynced to disk either).
>> My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure,
>>provided disks didn't crash ?
No, see my above reply.
-Dave
-----Original Message-----
From: JEVTIC, MARKO [mailto:MARKO.JEVTIC@fisglobal.com]
Sent: Tuesday, May 30, 2017 8:05 AM
To: users@kafka.apache.org
Subject: client recordmetadata meaning
Hi all,
I wasn't able to find in documentation firm agreement about Kafka message reply.
So, before going through the source code, I would like to ask a question:
If kafka client producer gets record meta data with a valid offset, do we consider that that message is indeed fsynced to the disk ?
My next related question is, if we have a single kafka server/broker, could we assume that all messages that we got a reply from Kafka client with a valid offset are successfully written to disk even in case of power failure, provided disks didn't crash ?
Thank you for your time.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.