Reboot

[Trevor Francis <tr...@tgrahamcapital.com>]

How does Flume handle a complete system reboot? Upon firing back up, does it immediately stream the contents of a log file…creating duplicates? I intend to deploy with the script with "taildir." Will it just start where it left off?


Trevor Francis

Re: Reboot

[phuduc nguyen <du...@pearson.com>]

Wait is that the exact command you¹re using? Sorry, maybe I confused you,
but just use the boolean as the parameter.

flume dump ŒtailDir(³ar/log/acc², ³.*², true)¹






On 4/18/12 7:47 AM, "Trevor Francis" <tr...@tgrahamcapital.com>
wrote:

> That makes sense, howeverŠ..
> 
> flume dump 'tailDir("ar/log/acc/", ".*", startFromEnd=true)'
> 
> Reads all the information from all the files in the directory, not just new
> information that was introduced to those files while flume was running. Why is
> this?
> 
> 
> Trevor Francis
> 
> 
> On Apr 18, 2012, at 8:20 AM, Nguyen, PhuDuc H wrote:
> 
>> It depends on the parameter you set.
>> 
>> tailDir("/path/to/dir", "fileAndRegex", startFromEnd)
>> 
>> If startFromEnd=false, the agent will re-read all files (from the beginning)
>> in the dir that match the regex. This creates duplicates on restart of an
>> agent. If startFromEnd=true, the agent will only read from the end of the
>> file (new incoming data). This setting can cause data loss.
>> 
>> You can pick your poison. Some people choose to allow duplicates because data
>> loss is generally unacceptable; then write a preprocessing mapreduce job that
>> filters out any duplicates. This m/r job becomes the 1st link in the chain of
>> your m/r analytics.
>> 
>> HTH,
>> Duc
>> 
>> 
>> 
>> -----Original Message-----
>> From: Trevor Francis [mailto:trevor.francis@tgrahamcapital.com]
>> Sent: Wed 4/18/2012 6:52 AM
>> To: flume-user@incubator.apache.org
>> Subject: Reboot
>> 
>> How does Flume handle a complete system reboot? Upon firing back up, does it
>> immediately stream the contents of a log file.creating duplicates? I intend
>> to deploy with the script with "taildir." Will it just start where it left
>> off?
>> 
>> 
>> Trevor Francis
>> 
>> 
>> <winmail.dat>
> 
> 

Re: Reboot

[Trevor Francis <tr...@tgrahamcapital.com>]

That makes sense, however…..

flume dump 'tailDir("ar/log/acc/", ".*", startFromEnd=true)'

Reads all the information from all the files in the directory, not just new information that was introduced to those files while flume was running. Why is this?


Trevor Francis


On Apr 18, 2012, at 8:20 AM, Nguyen, PhuDuc H wrote:

> It depends on the parameter you set. 
> 
> tailDir("/path/to/dir", "fileAndRegex", startFromEnd)
> 
> If startFromEnd=false, the agent will re-read all files (from the beginning) in the dir that match the regex. This creates duplicates on restart of an agent. If startFromEnd=true, the agent will only read from the end of the file (new incoming data). This setting can cause data loss. 
> 
> You can pick your poison. Some people choose to allow duplicates because data loss is generally unacceptable; then write a preprocessing mapreduce job that filters out any duplicates. This m/r job becomes the 1st link in the chain of your m/r analytics. 
> 
> HTH,
> Duc
> 
> 
> 
> -----Original Message-----
> From: Trevor Francis [mailto:trevor.francis@tgrahamcapital.com]
> Sent: Wed 4/18/2012 6:52 AM
> To: flume-user@incubator.apache.org
> Subject: Reboot
> 
> How does Flume handle a complete system reboot? Upon firing back up, does it immediately stream the contents of a log file.creating duplicates? I intend to deploy with the script with "taildir." Will it just start where it left off?
> 
> 
> Trevor Francis
> 
> 
> <winmail.dat>

RE: Reboot

["Nguyen, PhuDuc H" <Du...@Pearson.com>]

It depends on the parameter you set. 

tailDir("/path/to/dir", "fileAndRegex", startFromEnd)

If startFromEnd=false, the agent will re-read all files (from the beginning) in the dir that match the regex. This creates duplicates on restart of an agent. If startFromEnd=true, the agent will only read from the end of the file (new incoming data). This setting can cause data loss. 

You can pick your poison. Some people choose to allow duplicates because data loss is generally unacceptable; then write a preprocessing mapreduce job that filters out any duplicates. This m/r job becomes the 1st link in the chain of your m/r analytics. 

HTH,
Duc



-----Original Message-----
From: Trevor Francis [mailto:trevor.francis@tgrahamcapital.com]
Sent: Wed 4/18/2012 6:52 AM
To: flume-user@incubator.apache.org
Subject: Reboot
 
How does Flume handle a complete system reboot? Upon firing back up, does it immediately stream the contents of a log file.creating duplicates? I intend to deploy with the script with "taildir." Will it just start where it left off?


Trevor Francis