[jira] [Resolved] (CAMEL-15955) camel-core - API and SPI for Camel components to resume from offset

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-15955.
---------------------------------
    Resolution: Fixed

> camel-core - API and SPI for Camel components to resume from offset
> -------------------------------------------------------------------
>
>                 Key: CAMEL-15955
>                 URL: https://issues.apache.org/jira/browse/CAMEL-15955
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-core
>            Reporter: Claus Ibsen
>            Assignee: Otavio Rodolfo Piske
>            Priority: Major
>             Fix For: 3.12.0
>
>
> We need a API and SPI for components to optionally support some way of offset management. Meaning that a component is able to resume from a given offset to continue from last known state. For example if a Camel application was stopped/crashed, then it can restart and resume from last known offset.
> Something like Apache Kafka offset has.
> This benefits Camel in general, and Camel Kafka Connector also.
> An use-case is the famous big files. So the file component is reading a file, and process it line by line via a splitter. Then we would like to store an offset how far it has processed. And in case of crash/restart etc then it can continue from last known position. Today it would restart from top.
> Same for SQL,JDBC components with SQL ResultSet and so on.
> The offset management should support different quality levels and a SPI so different implementations can plugin. One could be to store to a plain file and for each step. And other can have a batch to store after N rows or N seconds. Or like in Kafka with every N seconds.
> Then each Camel component that can support this, can then implement the API and logic for handling this. For the file component we can maybe store an offset in the input stream, so when resuming we then forward the stream to that position.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)