You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Chandrashekhar Shaw <sh...@yahoo-inc.com> on 2012/07/23 13:52:20 UTC
Issues with flume while trying to transfer big size files [1GB
every hour]
Hi,
I am using flume for transferring logs from 20 agent boxes to 1 collector box. The file size which I am trying to transfer is of about 1 GB per hour. So every hour my agents run to transfer the log file generated during the last hour.
I am using DFO sink for file transfer.
But then I am facing following problems:
1) Some of the agents are not able to transfer the whole file even in one hour and the processes run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file completely.
I want to know
1) if somebody else has faced the same issues?
2) What could be the problem?
Thanks
Chandrashekhar Shaw
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Not at the moment, but there is already effort to support encryption in file channel:
https://issues.apache.org/jira/browse/FLUME-1424
Jarcec
On Wed, Aug 08, 2012 at 12:55:35PM +0530, Chandrashekhar Shaw wrote:
> Hi Harish,
>
> Thanks for quick response..
>
> No, Actually I did not try flume-ng. Flume-og was working good for me..
>
> Just curious to know does flume-ng supports encrypted channel ?
>
> Thanks
> Chandrashekhar
>
> On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
>
> > Chandrashekhar,
> >
> > Is there a specific reason you cannot use Flume NG. I believe that has already been suggested on this thread. Flume OG is no longer actively developed, and it is better to move to Flume NG - which had its second release a few weeks back - Flume v 1.2.0.
> >
> >
> > Thanks
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> >
> >> Hi
> >>
> >> I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
> >>
> >> It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
> >>
> >> * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
> >> * The process did not end even after 5/6 hours. (I had to manually kill it).
> >>
> >> As I know flume agent should continue sending the log once the collector is re-started. ???
> >> Is there something I am missing ??
> >>
> >> Thanks
> >> Chandrashekhar Shaw
> >>
> >>
> >>
> >>
> >> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> >>
> >> Hello,
> >>
> >> Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
> >>
> >> Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> >>
> >> This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> >>
> >> This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
> >>
> >> Regards,
> >> Harish
> >>
> >>
> >> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>> wrote:
> >> I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
> >>
> >> It there an available version which implements netty??
> >>
> >> Thanks
> >> Chandrashekhar Shaw
> >>
> >>
> >> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> >>
> >> Hi,
> >>
> >> Which one of the sources are you using? Maybe a flume source that
> >> implements something like
> >> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> >>
> >> Regards,
> >> Harish
> >>
> >>
> >> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> >> <sh...@yahoo-inc.com>>wrote:
> >>
> >> Hi,
> >>
> >> I am using flume for transferring logs from 20 agent boxes to 1 collector
> >> box. The file size which I am trying to transfer is of about 1 GB per hour.
> >> So every hour my agents run to transfer the log file generated during the
> >> last hour.
> >> I am using DFO sink for file transfer.
> >>
> >> But then I am facing following problems:
> >>
> >> 1) Some of the agents are not able to transfer the whole file even in one
> >> hour and the processes run continuously for hours.
> >> 2) Some of the agents stop within 5-6 minutes without sending the file
> >> completely.
> >>
> >> I want to know
> >>
> >> 1) if somebody else has faced the same issues?
> >> 2) What could be the problem?
> >>
> >> Thanks
> >> Chandrashekhar Shaw
> >>
> >>
> >
> >
>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Chandrashekhar Shaw <sh...@yahoo-inc.com>.
Hi Harish,
I meant encrypted channel.
Thanks
Shaw
On Aug 8, 2012, at 11:39 PM, Mike Percy wrote:
> Hi Chandrashekhar,
> When you say encrypted channel, can you be more specific about what you are
> referring to? i.e. encrypted on-disk or encrypted on the wire?
>
> Regards,
> Mike
>
> On Wed, Aug 8, 2012 at 4:05 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>wrote:
>
>> Or I would rather say "Does flume-ng provide encrypted channel ??"
>>
>> Thanks
>> Chandrashekhar
>> On Aug 8, 2012, at 12:55 PM, Chandrashekhar Shaw wrote:
>>
>>> Hi Harish,
>>>
>>> Thanks for quick response..
>>>
>>> No, Actually I did not try flume-ng. Flume-og was working good for me..
>>>
>>> Just curious to know does flume-ng supports encrypted channel ?
>>>
>>> Thanks
>>> Chandrashekhar
>>>
>>> On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
>>>
>>>> Chandrashekhar,
>>>>
>>>> Is there a specific reason you cannot use Flume NG. I believe that has
>> already been suggested on this thread. Flume OG is no longer actively
>> developed, and it is better to move to Flume NG - which had its second
>> release a few weeks back - Flume v 1.2.0.
>>>>
>>>>
>>>> Thanks
>>>> Hari
>>>>
>>>> --
>>>> Hari Shreedharan
>>>>
>>>>
>>>> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I was testing flume for log transfer in DFO mode. I was firing the
>> single shot command as cron every 10th minute from the agent box. So the
>> flume process starts every 10 minutes sends the logs to collector box and
>> ends.
>>>>>
>>>>> It was pushing the logs very well. To test the reliability I stopped
>> the collector and brought it back after 1 hour, what I see is
>>>>>
>>>>> * The flume process (started by the agent cron) kept on running in the
>> agent machine and not able to transfer the rest of the file to collector.
>>>>> * The process did not end even after 5/6 hours. (I had to manually
>> kill it).
>>>>>
>>>>> As I know flume agent should continue sending the log once the
>> collector is re-started. ???
>>>>> Is there something I am missing ??
>>>>>
>>>>> Thanks
>>>>> Chandrashekhar Shaw
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> Not too sure about Flume OG - I barely used it for a week before
>> shifting to NG. Would there be any particular reason you are using OG?
>>>>>
>>>>> Yes, you can take a look at SyslogTcpSource (in NG),
>> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
>>>>>
>>>>> This can be modified according to this Netty example (for asynchronous
>> large file streaming) :
>> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
>>>>>
>>>>> This is pretty much what I did when I needed a Flume HTTP source to
>> match my requirements.
>>>>>
>>>>> Regards,
>>>>> Harish
>>>>>
>>>>>
>>>>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <
>> shawcs@yahoo-inc.com<ma...@yahoo-inc.com>> wrote:
>>>>> I am using Flume - OG
>> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz.
>>>>>
>>>>> It there an available version which implements netty??
>>>>>
>>>>> Thanks
>>>>> Chandrashekhar Shaw
>>>>>
>>>>>
>>>>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Which one of the sources are you using? Maybe a flume source that
>>>>> implements something like
>>>>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
>>>>>
>>>>> Regards,
>>>>> Harish
>>>>>
>>>>>
>>>>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
>>>>> <sh...@yahoo-inc.com>>wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using flume for transferring logs from 20 agent boxes to 1
>> collector
>>>>> box. The file size which I am trying to transfer is of about 1 GB per
>> hour.
>>>>> So every hour my agents run to transfer the log file generated during
>> the
>>>>> last hour.
>>>>> I am using DFO sink for file transfer.
>>>>>
>>>>> But then I am facing following problems:
>>>>>
>>>>> 1) Some of the agents are not able to transfer the whole file even in
>> one
>>>>> hour and the processes run continuously for hours.
>>>>> 2) Some of the agents stop within 5-6 minutes without sending the file
>>>>> completely.
>>>>>
>>>>> I want to know
>>>>>
>>>>> 1) if somebody else has faced the same issues?
>>>>> 2) What could be the problem?
>>>>>
>>>>> Thanks
>>>>> Chandrashekhar Shaw
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Mike Percy <mp...@apache.org>.
Hi Chandrashekhar,
When you say encrypted channel, can you be more specific about what you are
referring to? i.e. encrypted on-disk or encrypted on the wire?
Regards,
Mike
On Wed, Aug 8, 2012 at 4:05 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>wrote:
> Or I would rather say "Does flume-ng provide encrypted channel ??"
>
> Thanks
> Chandrashekhar
> On Aug 8, 2012, at 12:55 PM, Chandrashekhar Shaw wrote:
>
> > Hi Harish,
> >
> > Thanks for quick response..
> >
> > No, Actually I did not try flume-ng. Flume-og was working good for me..
> >
> > Just curious to know does flume-ng supports encrypted channel ?
> >
> > Thanks
> > Chandrashekhar
> >
> > On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
> >
> >> Chandrashekhar,
> >>
> >> Is there a specific reason you cannot use Flume NG. I believe that has
> already been suggested on this thread. Flume OG is no longer actively
> developed, and it is better to move to Flume NG - which had its second
> release a few weeks back - Flume v 1.2.0.
> >>
> >>
> >> Thanks
> >> Hari
> >>
> >> --
> >> Hari Shreedharan
> >>
> >>
> >> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> >>
> >>> Hi
> >>>
> >>> I was testing flume for log transfer in DFO mode. I was firing the
> single shot command as cron every 10th minute from the agent box. So the
> flume process starts every 10 minutes sends the logs to collector box and
> ends.
> >>>
> >>> It was pushing the logs very well. To test the reliability I stopped
> the collector and brought it back after 1 hour, what I see is
> >>>
> >>> * The flume process (started by the agent cron) kept on running in the
> agent machine and not able to transfer the rest of the file to collector.
> >>> * The process did not end even after 5/6 hours. (I had to manually
> kill it).
> >>>
> >>> As I know flume agent should continue sending the log once the
> collector is re-started. ???
> >>> Is there something I am missing ??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>>
> >>>
> >>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> >>>
> >>> Hello,
> >>>
> >>> Not too sure about Flume OG - I barely used it for a week before
> shifting to NG. Would there be any particular reason you are using OG?
> >>>
> >>> Yes, you can take a look at SyslogTcpSource (in NG),
> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> >>>
> >>> This can be modified according to this Netty example (for asynchronous
> large file streaming) :
> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> >>>
> >>> This is pretty much what I did when I needed a Flume HTTP source to
> match my requirements.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <
> shawcs@yahoo-inc.com<ma...@yahoo-inc.com>> wrote:
> >>> I am using Flume - OG
> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz.
> >>>
> >>> It there an available version which implements netty??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> >>>
> >>> Hi,
> >>>
> >>> Which one of the sources are you using? Maybe a flume source that
> >>> implements something like
> >>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> >>> <sh...@yahoo-inc.com>>wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am using flume for transferring logs from 20 agent boxes to 1
> collector
> >>> box. The file size which I am trying to transfer is of about 1 GB per
> hour.
> >>> So every hour my agents run to transfer the log file generated during
> the
> >>> last hour.
> >>> I am using DFO sink for file transfer.
> >>>
> >>> But then I am facing following problems:
> >>>
> >>> 1) Some of the agents are not able to transfer the whole file even in
> one
> >>> hour and the processes run continuously for hours.
> >>> 2) Some of the agents stop within 5-6 minutes without sending the file
> >>> completely.
> >>>
> >>> I want to know
> >>>
> >>> 1) if somebody else has faced the same issues?
> >>> 2) What could be the problem?
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>
> >>
> >
>
>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Chandrashekhar Shaw <sh...@yahoo-inc.com>.
Or I would rather say "Does flume-ng provide encrypted channel ??"
Thanks
Chandrashekhar
On Aug 8, 2012, at 12:55 PM, Chandrashekhar Shaw wrote:
> Hi Harish,
>
> Thanks for quick response..
>
> No, Actually I did not try flume-ng. Flume-og was working good for me..
>
> Just curious to know does flume-ng supports encrypted channel ?
>
> Thanks
> Chandrashekhar
>
> On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
>
>> Chandrashekhar,
>>
>> Is there a specific reason you cannot use Flume NG. I believe that has already been suggested on this thread. Flume OG is no longer actively developed, and it is better to move to Flume NG - which had its second release a few weeks back - Flume v 1.2.0.
>>
>>
>> Thanks
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>>
>> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
>>
>>> Hi
>>>
>>> I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
>>>
>>> It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
>>>
>>> * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
>>> * The process did not end even after 5/6 hours. (I had to manually kill it).
>>>
>>> As I know flume agent should continue sending the log once the collector is re-started. ???
>>> Is there something I am missing ??
>>>
>>> Thanks
>>> Chandrashekhar Shaw
>>>
>>>
>>>
>>>
>>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
>>>
>>> Hello,
>>>
>>> Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
>>>
>>> Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
>>>
>>> This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
>>>
>>> This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
>>>
>>> Regards,
>>> Harish
>>>
>>>
>>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>> wrote:
>>> I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
>>>
>>> It there an available version which implements netty??
>>>
>>> Thanks
>>> Chandrashekhar Shaw
>>>
>>>
>>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
>>>
>>> Hi,
>>>
>>> Which one of the sources are you using? Maybe a flume source that
>>> implements something like
>>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
>>>
>>> Regards,
>>> Harish
>>>
>>>
>>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
>>> <sh...@yahoo-inc.com>>wrote:
>>>
>>> Hi,
>>>
>>> I am using flume for transferring logs from 20 agent boxes to 1 collector
>>> box. The file size which I am trying to transfer is of about 1 GB per hour.
>>> So every hour my agents run to transfer the log file generated during the
>>> last hour.
>>> I am using DFO sink for file transfer.
>>>
>>> But then I am facing following problems:
>>>
>>> 1) Some of the agents are not able to transfer the whole file even in one
>>> hour and the processes run continuously for hours.
>>> 2) Some of the agents stop within 5-6 minutes without sending the file
>>> completely.
>>>
>>> I want to know
>>>
>>> 1) if somebody else has faced the same issues?
>>> 2) What could be the problem?
>>>
>>> Thanks
>>> Chandrashekhar Shaw
>>>
>>>
>>
>>
>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Chandrashekhar Shaw <sh...@yahoo-inc.com>.
Hi Harish,
Thanks for quick response..
No, Actually I did not try flume-ng. Flume-og was working good for me..
Just curious to know does flume-ng supports encrypted channel ?
Thanks
Chandrashekhar
On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
> Chandrashekhar,
>
> Is there a specific reason you cannot use Flume NG. I believe that has already been suggested on this thread. Flume OG is no longer actively developed, and it is better to move to Flume NG - which had its second release a few weeks back - Flume v 1.2.0.
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
>
> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
>
>> Hi
>>
>> I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
>>
>> It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
>>
>> * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
>> * The process did not end even after 5/6 hours. (I had to manually kill it).
>>
>> As I know flume agent should continue sending the log once the collector is re-started. ???
>> Is there something I am missing ??
>>
>> Thanks
>> Chandrashekhar Shaw
>>
>>
>>
>>
>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
>>
>> Hello,
>>
>> Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
>>
>> Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
>>
>> This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
>>
>> This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
>>
>> Regards,
>> Harish
>>
>>
>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>> wrote:
>> I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
>>
>> It there an available version which implements netty??
>>
>> Thanks
>> Chandrashekhar Shaw
>>
>>
>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
>>
>> Hi,
>>
>> Which one of the sources are you using? Maybe a flume source that
>> implements something like
>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
>>
>> Regards,
>> Harish
>>
>>
>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
>> <sh...@yahoo-inc.com>>wrote:
>>
>> Hi,
>>
>> I am using flume for transferring logs from 20 agent boxes to 1 collector
>> box. The file size which I am trying to transfer is of about 1 GB per hour.
>> So every hour my agents run to transfer the log file generated during the
>> last hour.
>> I am using DFO sink for file transfer.
>>
>> But then I am facing following problems:
>>
>> 1) Some of the agents are not able to transfer the whole file even in one
>> hour and the processes run continuously for hours.
>> 2) Some of the agents stop within 5-6 minutes without sending the file
>> completely.
>>
>> I want to know
>>
>> 1) if somebody else has faced the same issues?
>> 2) What could be the problem?
>>
>> Thanks
>> Chandrashekhar Shaw
>>
>>
>
>
Re: Issues with flume while trying to transfer big size files
[1GB every hour]
Posted by Hari Shreedharan <hs...@cloudera.com>.
Chandrashekhar,
Is there a specific reason you cannot use Flume NG. I believe that has already been suggested on this thread. Flume OG is no longer actively developed, and it is better to move to Flume NG - which had its second release a few weeks back - Flume v 1.2.0.
Thanks
Hari
--
Hari Shreedharan
On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> Hi
>
> I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
>
> It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
>
> * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
> * The process did not end even after 5/6 hours. (I had to manually kill it).
>
> As I know flume agent should continue sending the log once the collector is re-started. ???
> Is there something I am missing ??
>
> Thanks
> Chandrashekhar Shaw
>
>
>
>
> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
>
> Hello,
>
> Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
>
> Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
>
> This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
>
> This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
>
> Regards,
> Harish
>
>
> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>> wrote:
> I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
>
> It there an available version which implements netty??
>
> Thanks
> Chandrashekhar Shaw
>
>
> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
>
> Hi,
>
> Which one of the sources are you using? Maybe a flume source that
> implements something like
> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
>
> Regards,
> Harish
>
>
> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> <sh...@yahoo-inc.com>>wrote:
>
> Hi,
>
> I am using flume for transferring logs from 20 agent boxes to 1 collector
> box. The file size which I am trying to transfer is of about 1 GB per hour.
> So every hour my agents run to transfer the log file generated during the
> last hour.
> I am using DFO sink for file transfer.
>
> But then I am facing following problems:
>
> 1) Some of the agents are not able to transfer the whole file even in one
> hour and the processes run continuously for hours.
> 2) Some of the agents stop within 5-6 minutes without sending the file
> completely.
>
> I want to know
>
> 1) if somebody else has faced the same issues?
> 2) What could be the problem?
>
> Thanks
> Chandrashekhar Shaw
>
>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Chandrashekhar Shaw <sh...@yahoo-inc.com>.
Hi
I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
* The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
* The process did not end even after 5/6 hours. (I had to manually kill it).
As I know flume agent should continue sending the log once the collector is re-started. ???
Is there something I am missing ??
Thanks
Chandrashekhar Shaw
On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
Hello,
Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
Regards,
Harish
On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <sh...@yahoo-inc.com>> wrote:
I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
It there an available version which implements netty??
Thanks
Chandrashekhar Shaw
On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
Hi,
Which one of the sources are you using? Maybe a flume source that
implements something like
http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
Regards,
Harish
On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
<sh...@yahoo-inc.com>>wrote:
Hi,
I am using flume for transferring logs from 20 agent boxes to 1 collector
box. The file size which I am trying to transfer is of about 1 GB per hour.
So every hour my agents run to transfer the log file generated during the
last hour.
I am using DFO sink for file transfer.
But then I am facing following problems:
1) Some of the agents are not able to transfer the whole file even in one
hour and the processes run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file
completely.
I want to know
1) if somebody else has faced the same issues?
2) What could be the problem?
Thanks
Chandrashekhar Shaw
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Harish Mandala <mv...@gmail.com>.
Hello,
Not too sure about Flume OG - I barely used it for a week before shifting
to NG. Would there be any particular reason you are using OG?
Yes, you can take a look at SyslogTcpSource (in NG),
https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
This can be modified according to this Netty example (for asynchronous
large file streaming) :
http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
This is pretty much what I did when I needed a Flume HTTP source to match
my requirements.
Regards,
Harish
On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw
<sh...@yahoo-inc.com>wrote:
> I am using Flume - OG
> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz
> .
>
> It there an available version which implements netty??
>
> Thanks
> Chandrashekhar Shaw
>
>
> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
>
> Hi,
>
> Which one of the sources are you using? Maybe a flume source that
> implements something like
> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
>
> Regards,
> Harish
>
>
> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> <sh...@yahoo-inc.com>wrote:
>
> Hi,
>
>
> I am using flume for transferring logs from 20 agent boxes to 1 collector
>
> box. The file size which I am trying to transfer is of about 1 GB per hour.
>
> So every hour my agents run to transfer the log file generated during the
>
> last hour.
>
> I am using DFO sink for file transfer.
>
>
> But then I am facing following problems:
>
>
> 1) Some of the agents are not able to transfer the whole file even in one
>
> hour and the processes run continuously for hours.
>
> 2) Some of the agents stop within 5-6 minutes without sending the file
>
> completely.
>
>
> I want to know
>
>
> 1) if somebody else has faced the same issues?
>
> 2) What could be the problem?
>
>
> Thanks
>
> Chandrashekhar Shaw
>
>
>
>
>
>
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Chandrashekhar Shaw <sh...@yahoo-inc.com>.
I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
It there an available version which implements netty??
Thanks
Chandrashekhar Shaw
On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
Hi,
Which one of the sources are you using? Maybe a flume source that
implements something like
http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
Regards,
Harish
On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
<sh...@yahoo-inc.com>>wrote:
Hi,
I am using flume for transferring logs from 20 agent boxes to 1 collector
box. The file size which I am trying to transfer is of about 1 GB per hour.
So every hour my agents run to transfer the log file generated during the
last hour.
I am using DFO sink for file transfer.
But then I am facing following problems:
1) Some of the agents are not able to transfer the whole file even in one
hour and the processes run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file
completely.
I want to know
1) if somebody else has faced the same issues?
2) What could be the problem?
Thanks
Chandrashekhar Shaw
Re: Issues with flume while trying to transfer big size files [1GB
every hour]
Posted by Harish Mandala <mv...@gmail.com>.
Hi,
Which one of the sources are you using? Maybe a flume source that
implements something like
http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
Regards,
Harish
On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
<sh...@yahoo-inc.com>wrote:
> Hi,
>
> I am using flume for transferring logs from 20 agent boxes to 1 collector
> box. The file size which I am trying to transfer is of about 1 GB per hour.
> So every hour my agents run to transfer the log file generated during the
> last hour.
> I am using DFO sink for file transfer.
>
> But then I am facing following problems:
>
> 1) Some of the agents are not able to transfer the whole file even in one
> hour and the processes run continuously for hours.
> 2) Some of the agents stop within 5-6 minutes without sending the file
> completely.
>
> I want to know
>
> 1) if somebody else has faced the same issues?
> 2) What could be the problem?
>
> Thanks
> Chandrashekhar Shaw
>
>
>