You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "67287685 (JIRA)" <ji...@apache.org> on 2018/05/03 06:12:00 UTC
[jira] [Updated] (FLUME-3233) log roll to a new name,the flume
collect repeat
[ https://issues.apache.org/jira/browse/FLUME-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
67287685 updated FLUME-3233:
----------------------------
Description:
we use the flume 1.7 to collect our log send to kafka.
our java application compressed the log at every day 00:00,the log name change aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.
this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the flume while not collect the new data,the log is become zip file,the data is lost.
To solve this problem，we Modify the java application and flume
1.modify java application
the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min -> aaa.20180503.log.gz,the flume while complete collect the data ,not lost.
2.modify flume 1.7
use inode judge the file uniqueness，not use inode + filename ,when use inode + filename，the flume while repeate collect when the aaa.log --> aaa.20180503.tmp,this two file have the same inode
Please help to see if our plan is feasible，thanks every .
was:
we use the flume 1.7 to collect our log send to kafka.
our java application compressed the log at every day 00:00,the log name change aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete in 1 second time.
this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the flume while not collect the new data,the log is become zip file,the data is lost.
To solve this problem，we Modify the java application and flume
1.modify java application
the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min -> aaa.20180503.log.gz,the flume while complete collect the data ,not lost.
2.modify flume 1.7
use inode judge the file uniqueness，not use inode + filename ,when use inode + filename，the flume while repeate collect when the aaa.log --> aaa.20180503.tmp,this two file have the same inode
Please help to see if our plan is feasible，thanks every .
> log roll to a new name,the flume collect repeat
> -----------------------------------------------
>
> Key: FLUME-3233
> URL: https://issues.apache.org/jira/browse/FLUME-3233
> Project: Flume
> Issue Type: Bug
> Affects Versions: 1.7.0
> Reporter: 67287685
> Priority: Major
>
> we use the flume 1.7 to collect our log send to kafka.
> our java application compressed the log at every day 00:00,the log name change aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.
> this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the flume while not collect the new data,the log is become zip file,the data is lost.
> To solve this problem，we Modify the java application and flume
> 1.modify java application
> the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min -> aaa.20180503.log.gz,the flume while complete collect the data ,not lost.
>
> 2.modify flume 1.7
> use inode judge the file uniqueness，not use inode + filename ,when use inode + filename，the flume while repeate collect when the aaa.log --> aaa.20180503.tmp,this two file have the same inode
>
> Please help to see if our plan is feasible，thanks every .
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org