You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Venkatesh B <ve...@gmail.com> on 2014/07/14 10:18:46 UTC
Flume custom source stopping after some time when started through
java program.
Hello,
I'm running flume in Cent os 6.3 version.
Apache flume version 1.3
I had written many FLUME CUSTOM SOURCES,
All sources are working fine, except two sources.
When I start flume i.e., flume-ng ...... command from shell it is working
fine, flume is collecting data continuously for 6hours I tested it for many
times about 10 times.
But when I start flume from java program, i.e., In one ".sh" file I will
write "flume-ng --conf path to conf folder -f flume configuration file path
-n agent" command, And from java program I will run this .sh file by
calling Runtime.getRuntime().exec(.sh file path).waitFor().
In flume custom source I have written logic for collecting data from web
and then I will write that data to channel processor. Here sink is hdfs.
Here the PROBLEM is flume is collecting data about 7 minutes and then the
not collecting when I start flume from java program.
When I search the running processes it is showing that flume is running but
my custom sources is not running.
I implemented logging for the flume custom source by logging what I came to
know is that flume custom source is not running at the same time no error
is caught.
Here flume custom source is stopping without any intimation of error.
What is the difference between running flume directly from shell and
through java program?
Why it is stopping did anyone face this problem?
What actually my flume custom source code looks is
in start() method I'm creating a thread and this thread will collect data.
Thanking you..
Re: Flume custom source stopping after some time when started through
java program.
Posted by SaravanaKumar TR <sa...@gmail.com>.
Thanks venkatesh .I already have this config in place.Flume is writing data
in hdfs till some point.
After some random point , it stops writing , I dont see any *.tmp file
created in hdfs but still flume agent is running.
I am not sure why i stops writing data..but still logfile produces data
continuously.
Is that any case where channel gets full or cannot push event to hdfs ?
On 14 July 2014 18:34, Venkatesh B <ve...@gmail.com> wrote:
> Hey Sarvana I think you seem to be at very basic level of start.
>
> Just try with the following configuration,
>
> channel_capacity=1000
> sink_rollInterval=300
> sink_rollCount=0
> sink_rollSize=0
> sink_batchSize=100
> channel_transactionCapacity=1000
>
> And the file which you are tailing it should add new data continuously.
>
>
>
> On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR <sa...@gmail.com>
> wrote:
>
>> No i am using flume provided source exec , just to tail -F <logfile>
>>
>> I start flume as below,
>>
>> /bin/flume-ng agent -c /d0/flume/conf -f
>> /d0/flume/conf/flume-conf.properties -n a1
>> -Dflume.root.logger=DEBUG,LOGFILE &
>>
>> I get logs in flume.log file.I could see flume agent running from ps.
>>
>> I use file/memory channel..Agents data for sometime but stops after that
>> , but it doesnt happen in certain interval as you mentioned 6 hrs , it
>> happens randomly.
>>
>> I think channel might gets full.NOt sure how to verfiy the same.Any idea?
>>
>> Please tell me how to enable logging to identify when source stops.
>>
>>
>> On 14 July 2014 16:04, Venkatesh B <ve...@gmail.com> wrote:
>>
>>> Hi sarvana,
>>>
>>> My flume agent is surely running, when I see in running processes ps -ef
>>> it is showing flume agent process is running.But my custom source is not
>>> running.
>>> I have implemented logging, if flume custom source is running then it
>>> will write to log files. I have written many log statements I know where it
>>> is stopping, but not know why it is stopping.
>>>
>>> My custom source works fine as when I run from shell directly but not
>>> from java program, then custom source is running for many hours.
>>> And also flume is working fine I have written other custom sources
>>> before and they are running for days continuously when I start from java
>>> program.
>>> Here I'm not understanding what is happening internally, when I start to
>>> run flume through java program then only my custom source stops working
>>> where as when I run through shell then it is working fine. Is your case is
>>> also similar.
>>>
>>> How are you starting flume, either by java program or by shell?
>>>
>>>
>>>
>>> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <
>>> saran0081986@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> My issue looks little similar to your.My exec source stops collecting
>>>> data after some time.
>>>>
>>>> But I like to know how you could say flume is running but ,your custom
>>>> source is not running.Let me know how you identify that.
>>>>
>>>> Because in running process "ps" i could see a single process i.e. flume
>>>> agent running.How we identity if source is running or not.
>>>>
>>>> Thanks,
>>>> Saravana
>>>>
>>>>
>>>> On 14 July 2014 13:48, Venkatesh B <ve...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm running flume in Cent os 6.3 version.
>>>>> Apache flume version 1.3
>>>>>
>>>>> I had written many FLUME CUSTOM SOURCES,
>>>>> All sources are working fine, except two sources.
>>>>>
>>>>> When I start flume i.e., flume-ng ...... command from shell it is
>>>>> working fine, flume is collecting data continuously for 6hours I tested it
>>>>> for many times about 10 times.
>>>>>
>>>>> But when I start flume from java program, i.e., In one ".sh" file I
>>>>> will write "flume-ng --conf path to conf folder -f flume configuration file
>>>>> path -n agent" command, And from java program I will run this .sh file by
>>>>> calling Runtime.getRuntime().exec(.sh file path).waitFor().
>>>>>
>>>>>
>>>>> In flume custom source I have written logic for collecting data from
>>>>> web and then I will write that data to channel processor. Here sink is hdfs.
>>>>>
>>>>> Here the PROBLEM is flume is collecting data about 7 minutes and then
>>>>> the not collecting when I start flume from java program.
>>>>>
>>>>> When I search the running processes it is showing that flume is
>>>>> running but my custom sources is not running.
>>>>> I implemented logging for the flume custom source by logging what I
>>>>> came to know is that flume custom source is not running at the same time no
>>>>> error is caught.
>>>>>
>>>>> Here flume custom source is stopping without any intimation of error.
>>>>>
>>>>> What is the difference between running flume directly from shell and
>>>>> through java program?
>>>>> Why it is stopping did anyone face this problem?
>>>>>
>>>>> What actually my flume custom source code looks is
>>>>> in start() method I'm creating a thread and this thread will collect
>>>>> data.
>>>>>
>>>>>
>>>>>
>>>>> Thanking you..
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Flume custom source stopping after some time when started through
java program.
Posted by Venkatesh B <ve...@gmail.com>.
Hey Sarvana I think you seem to be at very basic level of start.
Just try with the following configuration,
channel_capacity=1000
sink_rollInterval=300
sink_rollCount=0
sink_rollSize=0
sink_batchSize=100
channel_transactionCapacity=1000
And the file which you are tailing it should add new data continuously.
On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR <sa...@gmail.com>
wrote:
> No i am using flume provided source exec , just to tail -F <logfile>
>
> I start flume as below,
>
> /bin/flume-ng agent -c /d0/flume/conf -f
> /d0/flume/conf/flume-conf.properties -n a1
> -Dflume.root.logger=DEBUG,LOGFILE &
>
> I get logs in flume.log file.I could see flume agent running from ps.
>
> I use file/memory channel..Agents data for sometime but stops after that ,
> but it doesnt happen in certain interval as you mentioned 6 hrs , it
> happens randomly.
>
> I think channel might gets full.NOt sure how to verfiy the same.Any idea?
>
> Please tell me how to enable logging to identify when source stops.
>
>
> On 14 July 2014 16:04, Venkatesh B <ve...@gmail.com> wrote:
>
>> Hi sarvana,
>>
>> My flume agent is surely running, when I see in running processes ps -ef
>> it is showing flume agent process is running.But my custom source is not
>> running.
>> I have implemented logging, if flume custom source is running then it
>> will write to log files. I have written many log statements I know where it
>> is stopping, but not know why it is stopping.
>>
>> My custom source works fine as when I run from shell directly but not
>> from java program, then custom source is running for many hours.
>> And also flume is working fine I have written other custom sources before
>> and they are running for days continuously when I start from java program.
>> Here I'm not understanding what is happening internally, when I start to
>> run flume through java program then only my custom source stops working
>> where as when I run through shell then it is working fine. Is your case is
>> also similar.
>>
>> How are you starting flume, either by java program or by shell?
>>
>>
>>
>> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <saran0081986@gmail.com
>> > wrote:
>>
>>> Hi,
>>>
>>> My issue looks little similar to your.My exec source stops collecting
>>> data after some time.
>>>
>>> But I like to know how you could say flume is running but ,your custom
>>> source is not running.Let me know how you identify that.
>>>
>>> Because in running process "ps" i could see a single process i.e. flume
>>> agent running.How we identity if source is running or not.
>>>
>>> Thanks,
>>> Saravana
>>>
>>>
>>> On 14 July 2014 13:48, Venkatesh B <ve...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm running flume in Cent os 6.3 version.
>>>> Apache flume version 1.3
>>>>
>>>> I had written many FLUME CUSTOM SOURCES,
>>>> All sources are working fine, except two sources.
>>>>
>>>> When I start flume i.e., flume-ng ...... command from shell it is
>>>> working fine, flume is collecting data continuously for 6hours I tested it
>>>> for many times about 10 times.
>>>>
>>>> But when I start flume from java program, i.e., In one ".sh" file I
>>>> will write "flume-ng --conf path to conf folder -f flume configuration file
>>>> path -n agent" command, And from java program I will run this .sh file by
>>>> calling Runtime.getRuntime().exec(.sh file path).waitFor().
>>>>
>>>>
>>>> In flume custom source I have written logic for collecting data from
>>>> web and then I will write that data to channel processor. Here sink is hdfs.
>>>>
>>>> Here the PROBLEM is flume is collecting data about 7 minutes and then
>>>> the not collecting when I start flume from java program.
>>>>
>>>> When I search the running processes it is showing that flume is running
>>>> but my custom sources is not running.
>>>> I implemented logging for the flume custom source by logging what I
>>>> came to know is that flume custom source is not running at the same time no
>>>> error is caught.
>>>>
>>>> Here flume custom source is stopping without any intimation of error.
>>>>
>>>> What is the difference between running flume directly from shell and
>>>> through java program?
>>>> Why it is stopping did anyone face this problem?
>>>>
>>>> What actually my flume custom source code looks is
>>>> in start() method I'm creating a thread and this thread will collect
>>>> data.
>>>>
>>>>
>>>>
>>>> Thanking you..
>>>>
>>>>
>>>>
>>>
>>
>
Re: Flume custom source stopping after some time when started through
java program.
Posted by SaravanaKumar TR <sa...@gmail.com>.
No i am using flume provided source exec , just to tail -F <logfile>
I start flume as below,
/bin/flume-ng agent -c /d0/flume/conf -f
/d0/flume/conf/flume-conf.properties -n a1
-Dflume.root.logger=DEBUG,LOGFILE &
I get logs in flume.log file.I could see flume agent running from ps.
I use file/memory channel..Agents data for sometime but stops after that ,
but it doesnt happen in certain interval as you mentioned 6 hrs , it
happens randomly.
I think channel might gets full.NOt sure how to verfiy the same.Any idea?
Please tell me how to enable logging to identify when source stops.
On 14 July 2014 16:04, Venkatesh B <ve...@gmail.com> wrote:
> Hi sarvana,
>
> My flume agent is surely running, when I see in running processes ps -ef
> it is showing flume agent process is running.But my custom source is not
> running.
> I have implemented logging, if flume custom source is running then it will
> write to log files. I have written many log statements I know where it is
> stopping, but not know why it is stopping.
>
> My custom source works fine as when I run from shell directly but not from
> java program, then custom source is running for many hours.
> And also flume is working fine I have written other custom sources before
> and they are running for days continuously when I start from java program.
> Here I'm not understanding what is happening internally, when I start to
> run flume through java program then only my custom source stops working
> where as when I run through shell then it is working fine. Is your case is
> also similar.
>
> How are you starting flume, either by java program or by shell?
>
>
>
> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <sa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> My issue looks little similar to your.My exec source stops collecting
>> data after some time.
>>
>> But I like to know how you could say flume is running but ,your custom
>> source is not running.Let me know how you identify that.
>>
>> Because in running process "ps" i could see a single process i.e. flume
>> agent running.How we identity if source is running or not.
>>
>> Thanks,
>> Saravana
>>
>>
>> On 14 July 2014 13:48, Venkatesh B <ve...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm running flume in Cent os 6.3 version.
>>> Apache flume version 1.3
>>>
>>> I had written many FLUME CUSTOM SOURCES,
>>> All sources are working fine, except two sources.
>>>
>>> When I start flume i.e., flume-ng ...... command from shell it is
>>> working fine, flume is collecting data continuously for 6hours I tested it
>>> for many times about 10 times.
>>>
>>> But when I start flume from java program, i.e., In one ".sh" file I will
>>> write "flume-ng --conf path to conf folder -f flume configuration file path
>>> -n agent" command, And from java program I will run this .sh file by
>>> calling Runtime.getRuntime().exec(.sh file path).waitFor().
>>>
>>>
>>> In flume custom source I have written logic for collecting data from web
>>> and then I will write that data to channel processor. Here sink is hdfs.
>>>
>>> Here the PROBLEM is flume is collecting data about 7 minutes and then
>>> the not collecting when I start flume from java program.
>>>
>>> When I search the running processes it is showing that flume is running
>>> but my custom sources is not running.
>>> I implemented logging for the flume custom source by logging what I came
>>> to know is that flume custom source is not running at the same time no
>>> error is caught.
>>>
>>> Here flume custom source is stopping without any intimation of error.
>>>
>>> What is the difference between running flume directly from shell and
>>> through java program?
>>> Why it is stopping did anyone face this problem?
>>>
>>> What actually my flume custom source code looks is
>>> in start() method I'm creating a thread and this thread will collect
>>> data.
>>>
>>>
>>>
>>> Thanking you..
>>>
>>>
>>>
>>
>
Re: Flume custom source stopping after some time when started through
java program.
Posted by Venkatesh B <ve...@gmail.com>.
Hi sarvana,
My flume agent is surely running, when I see in running processes ps -ef it
is showing flume agent process is running.But my custom source is not
running.
I have implemented logging, if flume custom source is running then it will
write to log files. I have written many log statements I know where it is
stopping, but not know why it is stopping.
My custom source works fine as when I run from shell directly but not from
java program, then custom source is running for many hours.
And also flume is working fine I have written other custom sources before
and they are running for days continuously when I start from java program.
Here I'm not understanding what is happening internally, when I start to
run flume through java program then only my custom source stops working
where as when I run through shell then it is working fine. Is your case is
also similar.
How are you starting flume, either by java program or by shell?
On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <sa...@gmail.com>
wrote:
> Hi,
>
> My issue looks little similar to your.My exec source stops collecting data
> after some time.
>
> But I like to know how you could say flume is running but ,your custom
> source is not running.Let me know how you identify that.
>
> Because in running process "ps" i could see a single process i.e. flume
> agent running.How we identity if source is running or not.
>
> Thanks,
> Saravana
>
>
> On 14 July 2014 13:48, Venkatesh B <ve...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm running flume in Cent os 6.3 version.
>> Apache flume version 1.3
>>
>> I had written many FLUME CUSTOM SOURCES,
>> All sources are working fine, except two sources.
>>
>> When I start flume i.e., flume-ng ...... command from shell it is
>> working fine, flume is collecting data continuously for 6hours I tested it
>> for many times about 10 times.
>>
>> But when I start flume from java program, i.e., In one ".sh" file I will
>> write "flume-ng --conf path to conf folder -f flume configuration file path
>> -n agent" command, And from java program I will run this .sh file by
>> calling Runtime.getRuntime().exec(.sh file path).waitFor().
>>
>>
>> In flume custom source I have written logic for collecting data from web
>> and then I will write that data to channel processor. Here sink is hdfs.
>>
>> Here the PROBLEM is flume is collecting data about 7 minutes and then the
>> not collecting when I start flume from java program.
>>
>> When I search the running processes it is showing that flume is running
>> but my custom sources is not running.
>> I implemented logging for the flume custom source by logging what I came
>> to know is that flume custom source is not running at the same time no
>> error is caught.
>>
>> Here flume custom source is stopping without any intimation of error.
>>
>> What is the difference between running flume directly from shell and
>> through java program?
>> Why it is stopping did anyone face this problem?
>>
>> What actually my flume custom source code looks is
>> in start() method I'm creating a thread and this thread will collect data.
>>
>>
>>
>> Thanking you..
>>
>>
>>
>
Re: Flume custom source stopping after some time when started through
java program.
Posted by SaravanaKumar TR <sa...@gmail.com>.
Hi,
My issue looks little similar to your.My exec source stops collecting data
after some time.
But I like to know how you could say flume is running but ,your custom
source is not running.Let me know how you identify that.
Because in running process "ps" i could see a single process i.e. flume
agent running.How we identity if source is running or not.
Thanks,
Saravana
On 14 July 2014 13:48, Venkatesh B <ve...@gmail.com> wrote:
> Hello,
>
> I'm running flume in Cent os 6.3 version.
> Apache flume version 1.3
>
> I had written many FLUME CUSTOM SOURCES,
> All sources are working fine, except two sources.
>
> When I start flume i.e., flume-ng ...... command from shell it is working
> fine, flume is collecting data continuously for 6hours I tested it for many
> times about 10 times.
>
> But when I start flume from java program, i.e., In one ".sh" file I will
> write "flume-ng --conf path to conf folder -f flume configuration file path
> -n agent" command, And from java program I will run this .sh file by
> calling Runtime.getRuntime().exec(.sh file path).waitFor().
>
>
> In flume custom source I have written logic for collecting data from web
> and then I will write that data to channel processor. Here sink is hdfs.
>
> Here the PROBLEM is flume is collecting data about 7 minutes and then the
> not collecting when I start flume from java program.
>
> When I search the running processes it is showing that flume is running
> but my custom sources is not running.
> I implemented logging for the flume custom source by logging what I came
> to know is that flume custom source is not running at the same time no
> error is caught.
>
> Here flume custom source is stopping without any intimation of error.
>
> What is the difference between running flume directly from shell and
> through java program?
> Why it is stopping did anyone face this problem?
>
> What actually my flume custom source code looks is
> in start() method I'm creating a thread and this thread will collect data.
>
>
>
> Thanking you..
>
>
>