You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Yi, Gene" <Ge...@sc.com.INVALID> on 2018/06/28 00:42:36 UTC
RE: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in
a topic) ie consumers reading off a partition will always get records in
the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order
in which a producer sends data can be different from the order in which
they are received into the partitions. An idempotent producer can be used
to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy,
consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to do
> manage State / Transactional Scenarios such as Updating a Bank Account
> scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://www.sc.com/en/our-locations. Please refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered Bank’s Privacy Policy.
Re: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by Nicolas Guignard <ni...@gmail.com>.
Hi Malik,
I think it depends on the strategy you want for your producers and
consumers.
Actually one topic one partition you'll have partition ordered guaranteed
and consequently topic too.
However, if you have several partitions for one topic then the events in
your partitions will be ordered but it is not guaranteed at the topic
level. You sometimes decide to have several partitions for data replication
(leader and replica) but also because you want to be able to parallelize.
In the second case, your producers will shard the data over a set
of partitions/brokers, messages with the same key from the same producer
are delivered in order to your consumer.
Hope it helps.
Cheers
Le ven. 29 juin 2018 à 05:52, Malik, Shibha (GE Renewable Energy,
consultant) <Sh...@ge.com> a écrit :
> So say if there is only one consumer in a consumer group ( to make order
> guarantee scenario ) and it reads from only one of the partition at a time
> ( say the topic he is subscribing to is partitioned into 3 partitions ) ,
> the only use of putting data into other two partitions is :
>
> 1. Other consumers can consume from it.
> 2. It will help in buffering data which produces sends and will increase
> throughput and latency
>
> ??
>
> -----Original Message-----
> From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
> Sent: Thursday, June 28, 2018 10:59 AM
> To: users@kafka.apache.org
> Subject: EXT: Re: [External] Re: Is order of data is not maintained in
> Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios
> such as Updating a Bank Account scenarios etc
>
> Yes, that's how Kafka works - all partitions are read in parallel, but
> only one consumer form the same consumer group reads a partition at one
> time (a consumer may consume multiple partitions, but no two consumers from
> same group consume the same partition)
>
> Virgil.
>
> On 6/28/18, 7:45 PM, "Malik, Shibha (GE Renewable Energy, consultant)" <
> Shibha.Malik@ge.com> wrote:
>
> You mean we use multiple partitions for a topic say topic is event
> "Customer Account Registration" , we can have multiple consumers read from
> different partition at a time ( only one consumer per partition ) and can
> perform registration of different customers in parallel ?
>
>
> -----Original Message-----
> From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
> Sent: Thursday, June 28, 2018 1:05 AM
> To: users@kafka.apache.org
> Subject: EXT: Re: [External] Re: Is order of data is not maintained in
> Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios
> such as Updating a Bank Account scenarios etc
>
> Messages within the same partition are ordered. You don't need to use
> only one partition (unless you need global ordering) - you just need to use
> keys. E.g. if your key is the account number, then all operations done on
> the same account are ordered; if your key is the customer ID, all
> operations done by the same customer are ordered; etc. Just find a suitable
> key.
>
> Virgil.
>
> On 6/28/18, 4:12 AM, "Malik, Shibha (GE Renewable Energy, consultant)"
> <Sh...@ge.com> wrote:
>
> But then restricting a consumer to use only partition seems to be
> similar to traditional message queues ( except the persistence feature )
> and does not offer parallelism , isn’t it ?
>
> -----Original Message-----
> From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
> Sent: Wednesday, June 27, 2018 5:43 PM
> To: users@kafka.apache.org
> Subject: EXT: RE: [External] Re: Is order of data is not
> maintained in Kafka, Is Kafka not suitable to do manage State /
> Transactional Scenarios such as Updating a Bank Account scenarios etc
>
> I think you may use one partition only?
>
>
> ______________________________________________________________
>
> Gene Huijian Yi
> TS - FM - App
>
> Standard Chartered Global Business Services Co., Ltd.
> Phone: +862259806930
> Address: Standard Chartered Center, 3/F No.35 Xinhuanbei
> Road,
> TEDA, Tianjin - 300457
> E-mail: Gene.Yi@sc.com
> ______________________________________________________________
> Please consider the environment before printing this email
>
> -----Original Message-----
> From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
> Sent: Thursday, June 28, 2018 7:45 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Is order of data is not maintained in
> Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios
> such as Updating a Bank Account scenarios etc
>
> Hi,
>
> Kafka guarantees ordering only within a partition (not across
> partitions in a topic) ie consumers reading off a partition will always get
> records in the order they were *received into the partition*.
>
> Note that in some cases of retries by a non- idempotent producer,
> the order in which a producer sends data can be different from the order in
> which they are received into the partitions. An idempotent producer can be
> used to avoid this scenario.
>
>
>
>
> On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy,
> consultant), <Sh...@ge.com> wrote:
>
> > Is order of data is not maintained in Kafka, Is Kafka not
> suitable to
> > do manage State / Transactional Scenarios such as Updating a
> Bank
> > Account scenarios etc
> >
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Four-locations&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=cEcwatemcNjhYxixWObUK31MkSaNyes1LOpjrZMPrJk%3D&reserved=0.
> Please refer to
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Fprivacy-policy.html&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=vvInBt3TVOWqe1mMImHCSK3pVSRehR0xl3GZTe5FpGk%3D&reserved=0
> for Standard Chartered Bank’s Privacy Policy.
>
>
>
>
>
--
Software engineer
+61402750175
nicolas.guignard65@gmail.com
http://www.java-4all.com
RE: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com>.
So say if there is only one consumer in a consumer group ( to make order guarantee scenario ) and it reads from only one of the partition at a time ( say the topic he is subscribing to is partitioned into 3 partitions ) , the only use of putting data into other two partitions is :
1. Other consumers can consume from it.
2. It will help in buffering data which produces sends and will increase throughput and latency
??
-----Original Message-----
From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
Sent: Thursday, June 28, 2018 10:59 AM
To: users@kafka.apache.org
Subject: EXT: Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Yes, that's how Kafka works - all partitions are read in parallel, but only one consumer form the same consumer group reads a partition at one time (a consumer may consume multiple partitions, but no two consumers from same group consume the same partition)
Virgil.
On 6/28/18, 7:45 PM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
You mean we use multiple partitions for a topic say topic is event "Customer Account Registration" , we can have multiple consumers read from different partition at a time ( only one consumer per partition ) and can perform registration of different customers in parallel ?
-----Original Message-----
From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
Sent: Thursday, June 28, 2018 1:05 AM
To: users@kafka.apache.org
Subject: EXT: Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Messages within the same partition are ordered. You don't need to use only one partition (unless you need global ordering) - you just need to use keys. E.g. if your key is the account number, then all operations done on the same account are ordered; if your key is the customer ID, all operations done by the same customer are ordered; etc. Just find a suitable key.
Virgil.
On 6/28/18, 4:12 AM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Four-locations&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=cEcwatemcNjhYxixWObUK31MkSaNyes1LOpjrZMPrJk%3D&reserved=0. Please refer to https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Fprivacy-policy.html&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=vvInBt3TVOWqe1mMImHCSK3pVSRehR0xl3GZTe5FpGk%3D&reserved=0 for Standard Chartered Bank’s Privacy Policy.
Re: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by Virgil Palanciuc <vi...@adobe.com.INVALID>.
Yes, that's how Kafka works - all partitions are read in parallel, but only one consumer form the same consumer group reads a partition at one time (a consumer may consume multiple partitions, but no two consumers from same group consume the same partition)
Virgil.
On 6/28/18, 7:45 PM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
You mean we use multiple partitions for a topic say topic is event "Customer Account Registration" , we can have multiple consumers read from different partition at a time ( only one consumer per partition ) and can perform registration of different customers in parallel ?
-----Original Message-----
From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
Sent: Thursday, June 28, 2018 1:05 AM
To: users@kafka.apache.org
Subject: EXT: Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Messages within the same partition are ordered. You don't need to use only one partition (unless you need global ordering) - you just need to use keys. E.g. if your key is the account number, then all operations done on the same account are ordered; if your key is the customer ID, all operations done by the same customer are ordered; etc. Just find a suitable key.
Virgil.
On 6/28/18, 4:12 AM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Four-locations&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=cEcwatemcNjhYxixWObUK31MkSaNyes1LOpjrZMPrJk%3D&reserved=0. Please refer to https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Fprivacy-policy.html&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=vvInBt3TVOWqe1mMImHCSK3pVSRehR0xl3GZTe5FpGk%3D&reserved=0 for Standard Chartered Bank’s Privacy Policy.
RE: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com>.
You mean we use multiple partitions for a topic say topic is event "Customer Account Registration" , we can have multiple consumers read from different partition at a time ( only one consumer per partition ) and can perform registration of different customers in parallel ?
-----Original Message-----
From: Virgil Palanciuc [mailto:virgilp@adobe.com.INVALID]
Sent: Thursday, June 28, 2018 1:05 AM
To: users@kafka.apache.org
Subject: EXT: Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Messages within the same partition are ordered. You don't need to use only one partition (unless you need global ordering) - you just need to use keys. E.g. if your key is the account number, then all operations done on the same account are ordered; if your key is the customer ID, all operations done by the same customer are ordered; etc. Just find a suitable key.
Virgil.
On 6/28/18, 4:12 AM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Four-locations&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=cEcwatemcNjhYxixWObUK31MkSaNyes1LOpjrZMPrJk%3D&reserved=0. Please refer to https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Fprivacy-policy.html&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=vvInBt3TVOWqe1mMImHCSK3pVSRehR0xl3GZTe5FpGk%3D&reserved=0 for Standard Chartered Bank’s Privacy Policy.
Re: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by Virgil Palanciuc <vi...@adobe.com.INVALID>.
Messages within the same partition are ordered. You don't need to use only one partition (unless you need global ordering) - you just need to use keys. E.g. if your key is the account number, then all operations done on the same account are ordered; if your key is the customer ID, all operations done by the same customer are ordered; etc. Just find a suitable key.
Virgil.
On 6/28/18, 4:12 AM, "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com> wrote:
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Four-locations&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=cEcwatemcNjhYxixWObUK31MkSaNyes1LOpjrZMPrJk%3D&reserved=0. Please refer to https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.sc.com%2Fen%2Fprivacy-policy.html&data=02%7C01%7Cvirgilp%40adobe.com%7C0166adb561584ccb057f08d5dc944573%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636657451735238847&sdata=vvInBt3TVOWqe1mMImHCSK3pVSRehR0xl3GZTe5FpGk%3D&reserved=0 for Standard Chartered Bank’s Privacy Policy.
Re: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by Tushar Madhukar <tu...@gmail.com>.
If you use partitioning based on your client/customer ID, events for a
particular customer will go to a unique partition and hence will always be
ordered. This still allows processing events from different customers in
parallel. Will also support scaling out to some degree.
However, if you need event ordering across multiple customers, then above
won't work.
On Thu., 28 Jun. 2018, 11:20 am Yi, Gene, <Ge...@sc.com.invalid> wrote:
> Yes, that is correct.. I guess updating a Bank Account should be done in
> sequence not parallel.. the reason use Kafka is maybe the better throughput
> compare to other message queues?
>
>
>
> -----Original Message-----
> From: Malik, Shibha (GE Renewable Energy, consultant) [mailto:
> Shibha.Malik@ge.com]
> Sent: Thursday, June 28, 2018 9:13 AM
> To: users@kafka.apache.org
> Subject: RE: [External] Re: Is order of data is not maintained in Kafka,
> Is Kafka not suitable to do manage State / Transactional Scenarios such as
> Updating a Bank Account scenarios etc
>
> But then restricting a consumer to use only partition seems to be similar
> to traditional message queues ( except the persistence feature ) and does
> not offer parallelism , isn’t it ?
>
> -----Original Message-----
> From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
> Sent: Wednesday, June 27, 2018 5:43 PM
> To: users@kafka.apache.org
> Subject: EXT: RE: [External] Re: Is order of data is not maintained in
> Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios
> such as Updating a Bank Account scenarios etc
>
> I think you may use one partition only?
>
>
> ______________________________________________________________
> Please consider the environment before printing this email
>
> -----Original Message-----
> From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
> Sent: Thursday, June 28, 2018 7:45 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Is order of data is not maintained in Kafka, Is
> Kafka not suitable to do manage State / Transactional Scenarios such as
> Updating a Bank Account scenarios etc
>
> Hi,
>
> Kafka guarantees ordering only within a partition (not across partitions
> in a topic) ie consumers reading off a partition will always get records in
> the order they were *received into the partition*.
>
> Note that in some cases of retries by a non- idempotent producer, the
> order in which a producer sends data can be different from the order in
> which they are received into the partitions. An idempotent producer can be
> used to avoid this scenario.
>
>
>
>
> On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy,
> consultant), <Sh...@ge.com> wrote:
>
> > Is order of data is not maintained in Kafka, Is Kafka not suitable to
> > do manage State / Transactional Scenarios such as Updating a Bank
> > Account scenarios etc
> >
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https://www.sc.com/en/our-locations. Please
> refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered
> Bank’s Privacy Policy.
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https://www.sc.com/en/our-locations. Please
> refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered
> Bank’s Privacy Policy.
>
RE: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by "Yi, Gene" <Ge...@sc.com.INVALID>.
Yes, that is correct.. I guess updating a Bank Account should be done in sequence not parallel.. the reason use Kafka is maybe the better throughput compare to other message queues?
-----Original Message-----
From: Malik, Shibha (GE Renewable Energy, consultant) [mailto:Shibha.Malik@ge.com]
Sent: Thursday, June 28, 2018 9:13 AM
To: users@kafka.apache.org
Subject: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://www.sc.com/en/our-locations. Please refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered Bank’s Privacy Policy.
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://www.sc.com/en/our-locations. Please refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered Bank’s Privacy Policy.
RE: [External] Re: Is order of data is not maintained in Kafka, Is
Kafka not suitable to do manage State / Transactional Scenarios such as
Updating a Bank Account scenarios etc
Posted by "Malik, Shibha (GE Renewable Energy, consultant)" <Sh...@ge.com>.
But then restricting a consumer to use only partition seems to be similar to traditional message queues ( except the persistence feature ) and does not offer parallelism , isn’t it ?
-----Original Message-----
From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
Sent: Wednesday, June 27, 2018 5:43 PM
To: users@kafka.apache.org
Subject: EXT: RE: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
I think you may use one partition only?
______________________________________________________________
Gene Huijian Yi
TS - FM - App
Standard Chartered Global Business Services Co., Ltd.
Phone: +862259806930
Address: Standard Chartered Center, 3/F No.35 Xinhuanbei Road,
TEDA, Tianjin - 300457
E-mail: Gene.Yi@sc.com
______________________________________________________________
Please consider the environment before printing this email
-----Original Message-----
From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
Sent: Thursday, June 28, 2018 7:45 AM
To: users@kafka.apache.org
Subject: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc
Hi,
Kafka guarantees ordering only within a partition (not across partitions in a topic) ie consumers reading off a partition will always get records in the order they were *received into the partition*.
Note that in some cases of retries by a non- idempotent producer, the order in which a producer sends data can be different from the order in which they are received into the partitions. An idempotent producer can be used to avoid this scenario.
On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy, consultant), <Sh...@ge.com> wrote:
> Is order of data is not maintained in Kafka, Is Kafka not suitable to
> do manage State / Transactional Scenarios such as Updating a Bank
> Account scenarios etc
>
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https://www.sc.com/en/our-locations. Please refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered Bank’s Privacy Policy.