Camel marker files ".camelLock" not deleted

[Stephan Burkard <sb...@gmail.com>]

Hi Camel riders

I have a quite simple polling file consumer that runs fine on 2 instances.

from(fileEndpointUri)
.routeId(routeId)
.log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
.setBody(method(bomProcessor, "removeBomIfPresent"))
.to(jmsEndpointUri);


The fileUri options are:
- initialDelay=[random-range]
- delay=[configurable]
- antFilterCaseSensitive=false
- antInclude=*.xml
- preMove=processing
- delete=true
- moveFailed=../error

I generate a random start delay to avoid both consumers running at the same
time. However sometimes they overlap and want to process the same file.

In this case I see normal processing in the logfile of one instance and an
error in the log of the other instance. So it works perfectly. With one
small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
That makes the operation guys nervous.

Can I do something so that these marker files are deleted? Is this kind of
a misconfiguration of my file endpoint?

Thanks a lot
Stephan

Re: Camel marker files ".camelLock" not deleted

[Zoran Regvart <zo...@regvart.com>]

Hi Stephan,
try setting `readLockDeleteOrphanLockFiles` on the file endpoint. Also
you might want to use an idempotent repository and a shared resource
to isolate the consumers -- for instance a database table.

zoran

On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <sb...@gmail.com> wrote:
> Hi Camel riders
>
> I have a quite simple polling file consumer that runs fine on 2 instances.
>
> from(fileEndpointUri)
> .routeId(routeId)
> .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> .setBody(method(bomProcessor, "removeBomIfPresent"))
> .to(jmsEndpointUri);
>
>
> The fileUri options are:
> - initialDelay=[random-range]
> - delay=[configurable]
> - antFilterCaseSensitive=false
> - antInclude=*.xml
> - preMove=processing
> - delete=true
> - moveFailed=../error
>
> I generate a random start delay to avoid both consumers running at the same
> time. However sometimes they overlap and want to process the same file.
>
> In this case I see normal processing in the logfile of one instance and an
> error in the log of the other instance. So it works perfectly. With one
> small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
> That makes the operation guys nervous.
>
> Can I do something so that these marker files are deleted? Is this kind of
> a misconfiguration of my file endpoint?
>
> Thanks a lot
> Stephan



-- 
Zoran Regvart

Re: Camel marker files ".camelLock" not deleted

[Stephan Burkard <sb...@gmail.com>]

Hi Zoran and Claus

Thanks for the suggestions.

readLockDeleteOrphanLockFiles: I am on Camel 2.12, so this is currently no
option.
shuffle: dito

I will look up the coordinator in the Camel book. This is a good reason to
download the most current MEAP version :-)

Regards
Stephan


Claus Ibsen <cl...@gmail.com> schrieb am Mo., 13. März 2017 um
17:02 Uhr:

> Hi
>
> You can use shuffle=true to randomize the file order to spread the
> chance of clash.
>
> But when having competing consumers on the same file share you can end
> up with clashes.
> You can use a distributed coordinator which is covered in the 2nd ed
> of the Camel book in the transactional chapter. And as well in the
> upcoming chapter on clustering.
>
>
>
> On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <sb...@gmail.com>
> wrote:
> > Hi Camel riders
> >
> > I have a quite simple polling file consumer that runs fine on 2
> instances.
> >
> > from(fileEndpointUri)
> > .routeId(routeId)
> > .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> > .setBody(method(bomProcessor, "removeBomIfPresent"))
> > .to(jmsEndpointUri);
> >
> >
> > The fileUri options are:
> > - initialDelay=[random-range]
> > - delay=[configurable]
> > - antFilterCaseSensitive=false
> > - antInclude=*.xml
> > - preMove=processing
> > - delete=true
> > - moveFailed=../error
> >
> > I generate a random start delay to avoid both consumers running at the
> same
> > time. However sometimes they overlap and want to process the same file.
> >
> > In this case I see normal processing in the logfile of one instance and
> an
> > error in the log of the other instance. So it works perfectly. With one
> > small flaw: the marker files of Camel (whatever.camelLock) are not
> deleted.
> > That makes the operation guys nervous.
> >
> > Can I do something so that these marker files are deleted? Is this kind
> of
> > a misconfiguration of my file endpoint?
> >
> > Thanks a lot
> > Stephan
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Re: Camel marker files ".camelLock" not deleted

[Claus Ibsen <cl...@gmail.com>]

Hi

You can use shuffle=true to randomize the file order to spread the
chance of clash.

But when having competing consumers on the same file share you can end
up with clashes.
You can use a distributed coordinator which is covered in the 2nd ed
of the Camel book in the transactional chapter. And as well in the
upcoming chapter on clustering.



On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <sb...@gmail.com> wrote:
> Hi Camel riders
>
> I have a quite simple polling file consumer that runs fine on 2 instances.
>
> from(fileEndpointUri)
> .routeId(routeId)
> .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> .setBody(method(bomProcessor, "removeBomIfPresent"))
> .to(jmsEndpointUri);
>
>
> The fileUri options are:
> - initialDelay=[random-range]
> - delay=[configurable]
> - antFilterCaseSensitive=false
> - antInclude=*.xml
> - preMove=processing
> - delete=true
> - moveFailed=../error
>
> I generate a random start delay to avoid both consumers running at the same
> time. However sometimes they overlap and want to process the same file.
>
> In this case I see normal processing in the logfile of one instance and an
> error in the log of the other instance. So it works perfectly. With one
> small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
> That makes the operation guys nervous.
>
> Can I do something so that these marker files are deleted? Is this kind of
> a misconfiguration of my file endpoint?
>
> Thanks a lot
> Stephan



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2