Ordering in a Kafka Pipe line

[chinchu chinchu <ch...@gmail.com>]

Hello,
We  are building a data pipe line with the following semantics. We need to
maintain order till the last unit of work is done in this pipe line .We
cannot have a single partition  since that looses our ability to scale .
Looked at using partitioning keys ,but does that guarantee order in the
pipe line as such as opposed to between a topic and consumers?

produces > consumes enriches and produces >consumes ,does work and produces
> persistent store

Thanks,
Chinchu

Re: Ordering in a Kafka Pipe line

[Emmett Butler <em...@parsely.com>]

Kafka provides total ordering only within individual partitions. A topic
with multiple partitions is considered a "partial order", in which multiple
subsets of the topic are considered well-ordered but the topic as a whole
is not. The tradeoff between scalability via partitioning and message
ordering is fundamental to Kafka's design and unfortunately makes it
difficult to built Kafka-based pipelines that rely on strict ordering.

Partition keys are used to ensure that messages sharing a certain
characteristic will always go to the same partition. It's possible that you
could use this to your advantage given the ordering guarantees provided by
individual partitions, but you'd still be limited in your ability to scale
beyond a single partition for any given partition key.

On Fri, Jul 6, 2018 at 10:15 AM chinchu chinchu <ch...@gmail.com>
wrote:

> Hello,
> We  are building a data pipe line with the following semantics. We need to
> maintain order till the last unit of work is done in this pipe line .We
> cannot have a single partition  since that looses our ability to scale .
> Looked at using partitioning keys ,but does that guarantee order in the
> pipe line as such as opposed to between a topic and consumers?
>
> produces > consumes enriches and produces >consumes ,does work and produces
> > persistent store
>
> Thanks,
> Chinchu
>


-- 
Emmett Butler | Senior Software Engineer
<http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature>