flume agent on windows: locked tail files?

[Joshua Sumali <js...@kobo.com>]

Hi,

I'm having some difficulty with the 0.9.4 Windows agent and locked files.

I've been experimenting with tail() and tailDir() sources and noticed that:

-my application's log4net can't rotate tail'd files
-I can't rename/delete tail'd files manually either

It seems that flumeagent.exe is locking the file to any rename or deletes. Appends are OK.

When I kill the flumeagent.exe service/process, the files are back to normal.

Anybody have an idea why this might be happening, or a potential fix/workaround?

Thanks,
Josh

Re: flume agent on windows: locked tail files?

[Dan Everton <da...@iocaine.org>]

If I remember correctly, this is because the Java File API doesn't let you
specify the lock mode when opening a file (at least not until Java 7). So
Flume ends up opening all files with a lock mode that Windows treats as "no
one else can move/delete this file". The fix was to use the new API in Java
7 but I think the tail and tailDir plugins have been removed completely in
Flume NG.

On Linux this isn't really a problem as the file locking semantics allow
files to be moved/deleted while another process has them open. Never did
fine a work around for Windows apart from restarting the Flume agent
process on logrotate.

Cheers,
Dan

Re: flume agent on windows: locked tail files?

[Søren <sd...@syntonetic.com>]

Hi Josh

We experienced the same and ended up sending log lines directly to the 
local flume agent via thrift. We will do something similar with flume-ng 
as soon as it works on windows.

kind regards
Søren

On 26/06/2012 00:19, Joshua Sumali wrote:
>
> Hi,
>
> I’m having some difficulty with the 0.9.4 Windows agent and locked files.
>
> I’ve been experimenting with tail() and tailDir() sources and noticed 
> that:
>
> -my application’s log4net can’t rotate tail’d files
>
> -I can’t rename/delete tail’d files manually either
>
> It seems that flumeagent.exe is locking the file to any rename or 
> deletes. Appends are OK.
>
> When I kill the flumeagent.exe service/process, the files are back to 
> normal.
>
> Anybody have an idea why this might be happening, or a potential 
> fix/workaround?
>
> Thanks,
>
> Josh
>