Single instance streaming from sftp

[dsemenov <sd...@gmail.com>]

Hello community.

I have to implement a solution, where two jvm instances run the same camel
route, but only one (leader) has to complete it.

The route should be executed by scheduler (timer).

The route execution (in case that jvm instance is leader) should retrieve
file via sftp, and process each line, and then send it to other endpoint
(sftp as well).
The file could be very big, so streaming should be applied to avid memory
consumption. New line separator should be used as a tokenizer.

We implemented some route to acquire leadership, which sets the Body = true
in case of success.

so we implement our single instance executions like:

<from uri="timer://timer.... />
 <setBody>
  <groovy>InetAddress.getLocalHost().getHostName()</groovy>
 </setBody>
 <to uri="dirrect:acquireLeadership" />
 <choice>
  <when>
   <simple>body</simple>
   <log message="instance is leader, processing" />
   <to uri="direct:process" />
  </when>
  <otherwise>
   <log message="failed to acquire leadership" />
  </otherwise>
 </choice>
</route>

My question is how can i organize direct:process sub-route to consume and
stream sftp files.

I was looking on <stream/> and <enrich/> endpoints, but i cannot figure out
how can i organize processing sub-routes, because i need another consumer
for <stream/> like

<camel:from uri="stream:file?fileName=/path/to/file/downloaded/from/sftp"/>

Maybe i can organize cross-process locking using sftp consumer itself, i.e
lock files, but i think it's impossible, because each instance has it's own
filesystem.

Additional info.
- We use camel 2.8.0
- We cannot use zookeeper and other tools for cross-process synchronization 




--
View this message in context: http://camel.465427.n5.nabble.com/Single-instance-streaming-from-sftp-tp5732771.html
Sent from the Camel - Users mailing list archive at Nabble.com.