Streaming processing in real life scenario

[po...@gmx.com]

Hello Guys,

I’m trying to understand streaming processing in real life scenario. 
Yeah, we know systems produce data, these huge amount of data are sent to Kafka (or other platform), Kafka redistribute them to topics (sometimes enrich them with other information), ‘consumers’ use them in some way. Reports, frauds, e-commerce recommendations… blah, blah blah…

But this is quite simple scenario – one system sends, other receives. No interaction. Of course we have this ‘security’, ‘no single database’, ‘scalability’… blah, blah, blah

But what if we need to design some interaction. For instance hotel booking. We can have several booking systems (several operators) – reservation made by one system for a room can is visible to others (others read content of topics). But when some system place reservation it needs information weather room is available or not and block other potential reservations to avoid conflicts. 

Looks like still one, central place with ‘traditional’ API is required? At least to manage essential information.
Similar information with e-commerce orders, stock availability and so on.
What’s the best system structure for ‘stream processing’ in such situations? 😊

Regards,

Mike

Re: [EXTERNAL] Streaming processing in real life scenario

["Tauzell, Dave" <Da...@surescripts.com>]

I don’t think Streaming is a good fit for those use cases.  It is best for asynchronous processes.  For example, once something is hotel room is booked and paid then send a message to update the rewards account.   But if you need to gate access to something (like if a room is booked or not) then you need a more traditional database or single owner.

-Dave

From: podunk@gmx.com <po...@gmx.com>
Date: Monday, October 3, 2022 at 6:15 AM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: [EXTERNAL] Streaming processing in real life scenario

Hello Guys,

I’m trying to understand streaming processing in real life scenario.
Yeah, we know systems produce data, these huge amount of data are sent to Kafka (or other platform), Kafka redistribute them to topics (sometimes enrich them with other information), ‘consumers’ use them in some way. Reports, frauds, e-commerce recommendations… blah, blah blah…

But this is quite simple scenario – one system sends, other receives. No interaction. Of course we have this ‘security’, ‘no single database’, ‘scalability’… blah, blah, blah

But what if we need to design some interaction. For instance hotel booking. We can have several booking systems (several operators) – reservation made by one system for a room can is visible to others (others read content of topics). But when some system place reservation it needs information weather room is available or not and block other potential reservations to avoid conflicts.

Looks like still one, central place with ‘traditional’ API is required? At least to manage essential information.
Similar information with e-commerce orders, stock availability and so on.
What’s the best system structure for ‘stream processing’ in such situations? 😊

Regards,

Mike
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Streaming processing in real life scenario

[Henrik Pang <he...@simplemail.co.in>]

we run this stream processing in Recommender Systems.
It has at least two advances reading from streaming:

1. fast feedback to recall model. the model should have the ability to 
know the results for given recommended object, so streaming feedback is 
suitable here.
2. easy scaling. when the deployed ML models for prediction have reached 
the max throughput, a new instance will be scaled up which should read 
the same dataset from streaming store.

Thanks


podunk@gmx.com wrote:
> What’s the best system structure for ‘stream processing’ in such situations? 😊

-- 
Simple Mail
https://simplemail.co.in/