You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Chandrashekhar Shaw <sh...@yahoo-inc.com> on 2012/07/24 09:07:10 UTC

Issues with flume while trying to transfer big size files [~1GB per hour]

Hi,

I am using flume for transferring logs from 20 agent boxes to 1 collector box. The file size which I am trying to transfer is of about 1 GB per hour. So every hour my agents run to transfer the log file generated during the last hour.
I am using DFO sink for file transfer.

But then I am facing following problems:

1) Some of the agents are not able to transfer the whole file even in one hour and the processes run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file completely.

I want to know 

1) if somebody else has faced the same issues?
2) What could be the problem?

Thanks
Chandrashekhar Shaw