Mongodb Persistent tail tracking: How to chose an increasing field for tracking

[jpeschke <pe...@etone.de>]

Hello,
Perhaps this is more a MongoDB issue, but maybe somebody has an idea:

The Camel MongoDB endpoint supports the persistent tail tracking feature
(which means that it stores the value of an arbitrary increasing field in a
document to reset the tailable cursor to this document when
restarting/restting the cursor). The field can be for example a timestamp.

Now my problem is:
How do I get such an "constantly increasing field" if I haven't any? MongoDB
hasn't something like an "auto increment" data type, so currently, I use a
counters collection which generates sequences of increasing ids (as
suggested in
https://docs.mongodb.com/v3.0/tutorial/create-an-auto-incrementing-field/#auto-increment-counters-collection)

However, this leads to some race conditions in a
multi-threaded/multi-machine environment, as the process of "Sequence ID
generation" and "insetion" is not atomic:

- Consider two servers (A and B), every one inserts an update. A gets the
first sequence number (e.g. 1), but B inserts it's document (with sequence
number 2) earlier, so the natural (= insertion) order in the capped
collection will now be "2 - 1".
- Now, as the tailable cursor consumes the capped collections in natural
order, it will start consuming the document with id 2. If the cursor is
reset/regenerated at exactly this point, the persistent tail tracker will
store "2" as the last processed document, although "1" was never processed,
so 1 will be lost :(.

Sounds academic, I know, but leads to some annoying errors in out
application :(

Thank you for any ideas.

Best regards,
Joerg





--
View this message in context: http://camel.465427.n5.nabble.com/Mongodb-Persistent-tail-tracking-How-to-chose-an-increasing-field-for-tracking-tp5787124.html
Sent from the Camel - Users mailing list archive at Nabble.com.