Re: [External] Re: Is order of data is not maintained in Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios such as Updating a Bank Account scenarios etc

[Tushar Madhukar <tu...@gmail.com>]

If you use partitioning based on your client/customer ID, events for a
particular customer will go to a unique partition and hence will always be
ordered. This still allows processing events from different customers in
parallel. Will also support scaling out to some degree.

However, if you need event ordering across multiple customers, then above
won't work.

On Thu., 28 Jun. 2018, 11:20 am Yi, Gene, <Ge...@sc.com.invalid> wrote:

> Yes, that is correct.. I guess updating a Bank Account should be done in
> sequence not parallel.. the reason use Kafka is maybe the better throughput
> compare to other message queues?
>
>
>
> -----Original Message-----
> From: Malik, Shibha (GE Renewable Energy, consultant) [mailto:
> Shibha.Malik@ge.com]
> Sent: Thursday, June 28, 2018 9:13 AM
> To: users@kafka.apache.org
> Subject: RE: [External] Re: Is order of data is not maintained in Kafka,
> Is Kafka not suitable to do manage State / Transactional Scenarios such as
> Updating a Bank Account scenarios etc
>
> But then restricting a consumer to use only partition seems to be similar
> to traditional message queues ( except the persistence feature ) and does
> not offer parallelism , isn’t it ?
>
> -----Original Message-----
> From: Yi, Gene [mailto:Gene.Yi@sc.com.INVALID]
> Sent: Wednesday, June 27, 2018 5:43 PM
> To: users@kafka.apache.org
> Subject: EXT: RE: [External] Re: Is order of data is not maintained in
> Kafka, Is Kafka not suitable to do manage State / Transactional Scenarios
> such as Updating a Bank Account scenarios etc
>
> I think you may use one partition only?
>
>
> ______________________________________________________________
> Please consider the environment before printing this email
>
> -----Original Message-----
> From: Tushar Madhukar [mailto:tushar.madhukar@gmail.com]
> Sent: Thursday, June 28, 2018 7:45 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Is order of data is not maintained in Kafka, Is
> Kafka not suitable to do manage State / Transactional Scenarios such as
> Updating a Bank Account scenarios etc
>
> Hi,
>
> Kafka guarantees ordering only within a partition (not across partitions
> in a topic) ie consumers reading off a partition will always get records in
> the order they were *received into the partition*.
>
> Note that in some cases of retries by a non- idempotent producer, the
> order in which a producer sends data can be different from the order in
> which they are received into the partitions. An idempotent producer can be
> used to avoid this scenario.
>
>
>
>
> On Thu., 28 Jun. 2018, 8:33 am Malik, Shibha (GE Renewable Energy,
> consultant), <Sh...@ge.com> wrote:
>
> > Is order of data is not maintained in Kafka, Is Kafka not suitable to
> > do manage State / Transactional Scenarios such as Updating a Bank
> > Account scenarios etc
> >
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https://www.sc.com/en/our-locations. Please
> refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered
> Bank’s Privacy Policy.
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https://www.sc.com/en/our-locations. Please
> refer to https://www.sc.com/en/privacy-policy.html for Standard Chartered
> Bank’s Privacy Policy.
>