You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Carlotta Hicks <Ca...@sas.com> on 2015/01/23 05:58:45 UTC
When to stop an agent with a Spool Directory source
I am a new Flume user. I have a java application where I would like to start a spool directory source, file channel and hbase sink.
After I start the components and copy a file into the spooldir directory, programmatically, how do I know when to
stop the components?
RE: When to stop an agent with a Spool Directory source
Posted by Carlotta Hicks <Ca...@sas.com>.
Thanks.
From: Mohit Durgapal [mailto:durgapalmohit@gmail.com]
Sent: Friday, January 23, 2015 1:10 AM
To: user
Subject: Re: When to stop an agent with a Spool Directory source
Currently I am using cloudera to start/stop flume agents.
But earlier I used to provide the source& sink configuration while starting the sink from terminal with a command like below:
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Regards
Mohit
On Fri, Jan 23, 2015 at 11:01 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
How/where are you starting your sources and sinks?
From: Mohit Durgapal [mailto:durgapalmohit@gmail.com<ma...@gmail.com>]
Sent: Friday, January 23, 2015 12:20 AM
To: user
Subject: Re: When to stop an agent with a Spool Directory source
I am not sure about your use case but I am writing about my experiences with flume.
Generally you would use flume for streaming data and will not stop the components ever. You have source & sinks running all the time so that as soon as an application writes into the spool directory, the source reads it(as per the flume agent configuration) and puts it the in the channel from where sink consumes it.
Otherwise, I can't think of anything other than a shell script to run periodically to start-stop the agent.
Regards
Mohit
On Fri, Jan 23, 2015 at 10:28 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
I am a new Flume user. I have a java application where I would like to start a spool directory source, file channel and hbase sink.
After I start the components and copy a file into the spooldir directory, programmatically, how do I know when to
stop the components?
Re: When to stop an agent with a Spool Directory source
Posted by Mohit Durgapal <du...@gmail.com>.
correction:
instead of "while starting the sink" i meant "while starting the agent".
On Fri, Jan 23, 2015 at 11:39 AM, Mohit Durgapal <du...@gmail.com>
wrote:
>
> Currently I am using cloudera to start/stop flume agents.
>
> But earlier I used to provide the source& sink configuration while
> starting the sink from terminal with a command like below:
>
> bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
>
> Regards
> Mohit
>
> On Fri, Jan 23, 2015 at 11:01 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
>
>> How/where are you starting your sources and sinks?
>>
>>
>>
>> *From:* Mohit Durgapal [mailto:durgapalmohit@gmail.com]
>> *Sent:* Friday, January 23, 2015 12:20 AM
>> *To:* user
>> *Subject:* Re: When to stop an agent with a Spool Directory source
>>
>>
>>
>> I am not sure about your use case but I am writing about my experiences
>> with flume.
>> Generally you would use flume for streaming data and will not stop the
>> components ever. You have source & sinks running all the time so that as
>> soon as an application writes into the spool directory, the source reads
>> it(as per the flume agent configuration) and puts it the in the channel
>> from where sink consumes it.
>>
>> Otherwise, I can't think of anything other than a shell script to run
>> periodically to start-stop the agent.
>>
>> Regards
>>
>> Mohit
>>
>>
>>
>> On Fri, Jan 23, 2015 at 10:28 AM, Carlotta Hicks <Ca...@sas.com>
>> wrote:
>>
>>
>>
>> I am a new Flume user. I have a java application where I would like to
>> start a spool directory source, file channel and hbase sink.
>>
>>
>>
>> After I start the components and copy a file into the spooldir directory,
>> programmatically, how do I know when to
>>
>> stop the components?
>>
>>
>>
>>
>>
>
>
Re: When to stop an agent with a Spool Directory source
Posted by Mohit Durgapal <du...@gmail.com>.
Currently I am using cloudera to start/stop flume agents.
But earlier I used to provide the source& sink configuration while starting
the sink from terminal with a command like below:
bin/flume-ng agent --conf conf --conf-file example.conf --name a1
-Dflume.root.logger=INFO,console
Regards
Mohit
On Fri, Jan 23, 2015 at 11:01 AM, Carlotta Hicks <Ca...@sas.com>
wrote:
> How/where are you starting your sources and sinks?
>
>
>
> *From:* Mohit Durgapal [mailto:durgapalmohit@gmail.com]
> *Sent:* Friday, January 23, 2015 12:20 AM
> *To:* user
> *Subject:* Re: When to stop an agent with a Spool Directory source
>
>
>
> I am not sure about your use case but I am writing about my experiences
> with flume.
> Generally you would use flume for streaming data and will not stop the
> components ever. You have source & sinks running all the time so that as
> soon as an application writes into the spool directory, the source reads
> it(as per the flume agent configuration) and puts it the in the channel
> from where sink consumes it.
>
> Otherwise, I can't think of anything other than a shell script to run
> periodically to start-stop the agent.
>
> Regards
>
> Mohit
>
>
>
> On Fri, Jan 23, 2015 at 10:28 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
>
>
>
> I am a new Flume user. I have a java application where I would like to
> start a spool directory source, file channel and hbase sink.
>
>
>
> After I start the components and copy a file into the spooldir directory,
> programmatically, how do I know when to
>
> stop the components?
>
>
>
>
>
RE: When to stop an agent with a Spool Directory source
Posted by Carlotta Hicks <Ca...@sas.com>.
How/where are you starting your sources and sinks?
From: Mohit Durgapal [mailto:durgapalmohit@gmail.com]
Sent: Friday, January 23, 2015 12:20 AM
To: user
Subject: Re: When to stop an agent with a Spool Directory source
I am not sure about your use case but I am writing about my experiences with flume.
Generally you would use flume for streaming data and will not stop the components ever. You have source & sinks running all the time so that as soon as an application writes into the spool directory, the source reads it(as per the flume agent configuration) and puts it the in the channel from where sink consumes it.
Otherwise, I can't think of anything other than a shell script to run periodically to start-stop the agent.
Regards
Mohit
On Fri, Jan 23, 2015 at 10:28 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
I am a new Flume user. I have a java application where I would like to start a spool directory source, file channel and hbase sink.
After I start the components and copy a file into the spooldir directory, programmatically, how do I know when to
stop the components?
Re: When to stop an agent with a Spool Directory source
Posted by Mohit Durgapal <du...@gmail.com>.
I am not sure about your use case but I am writing about my experiences
with flume.
Generally you would use flume for streaming data and will not stop the
components ever. You have source & sinks running all the time so that as
soon as an application writes into the spool directory, the source reads
it(as per the flume agent configuration) and puts it the in the channel
from where sink consumes it.
Otherwise, I can't think of anything other than a shell script to run
periodically to start-stop the agent.
Regards
Mohit
On Fri, Jan 23, 2015 at 10:28 AM, Carlotta Hicks <Ca...@sas.com>
wrote:
>
>
> I am a new Flume user. I have a java application where I would like to
> start a spool directory source, file channel and hbase sink.
>
>
>
> After I start the components and copy a file into the spooldir directory,
> programmatically, how do I know when to
>
> stop the components?
>
>
>