what are the alternative for IgniteQueue for FIFO transactional, reliable, low-latency messaging

[Deepa Kolwalkar <de...@tcs.com>]

We have a requirement as follows:
Large streams of Time-ordered data received form over a TCP source to be 
processed in a time-ordered (cannot do parallel processing of the records) 
manner. 
The raw data or processed data does not need to be retained/cached in the 
Ignite layer.
The processed records need to be inserted into a DB.

The SLA is 
a) Lowest Latency b/w the stream ingestion till DB. 
b) Reliability : cannot afford losing any record/packet from the stream

In view of the above SLAs, would appreciate any feedback/suggestions on 
the following :

Q1) what would be a better choice for Stream Ingestion from TCP source : 
Ignite SocketStreamer OR Kafka Streamer ? 

Q2) since the records are time-ordered and need to be processed in the 
same manner,  we need a FIFO Queue in which the Streamer can inject the 
raw records, which can be picked up by a ClusterSingletonService which can 
process the raw records in a time-ordered manner & then insert the 
processed record in a write-through Cache to DB. 
Pls note that to ensure Reliability , the take() of each record from the 
FIFO Queue by the ClusterSingletonService till it writes the processed 
record to the Cache --> needs to be completed within single User 
Transaction, such that in the event of any failure/node-crash/etc while 
processing the record, it must rollback the transaction and NOT remove the 
raw record from the FIFO Queue.
I tried IgniteQueue, but it does not support User Transaction. 
What would be the best alternative for the FIFO Queue implementation ?

Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: what are the alternative for IgniteQueue for FIFO transactional, reliable, low-latency messaging

[dkarachentsev <dk...@gridgain.com>]

Hi,

I think the best way here would be to read items directly from kafka,
process and store in cache and rememeber in another cache kafka stream
offset. If node crashes, your service could start from the last point
(offset).

Thanks!
-Dmitry



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/