You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by mahendran m <ma...@hotmail.com> on 2014/11/09 16:48:18 UTC

Perfromacnce issue in file channel

Hi All ,I have tested the file channel and file channel performance. below the code snippet to generate 1000 event per sec 
while (watch.Elapsed.TotalSeconds < sec){	if (counter <= 1000)	{		uuid = uuid + 1;
		var text = HttpUtility.UrlEncode("HI , How are you", Encoding.UTF8);
		var flumeEvent = new ThriftFlumeEvent();		flumeEvent.Headers = new Dictionary<string, string>() { { "Host", "http://www.bing.com/" } };		flumeEvent.Body = Encoding.ASCII.GetBytes(text);		Client.append(flumeEvent);		counter += 1;	}}
i succeed in sending the 1000 event per sec when using memory channel , but in case if i used file channel  i can  only reach upto 40 to 50 events per second
config for memory channel# Use memory channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000000a1.channels.c1.transactionCapacity = 1000a1.channels.c1.byteCapacityBufferPercentage = 10a1.channels.c1.byteCapacity = 5368709120

config for file channel
#using the file channela1.channels.c1.type = filea1.channels.c1.capacity = 500000a1.channels.c1.transactionCapacity = 10000a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = F:/.checkPoint
Can anyone explain why file channel is this much channel ? . if i used file channel in server for directly logging each request is it better option as my server receive 100 to 200 request per second
Regards,Mahendran

 		 	   		   		 	   		  

RE: File channel data lost

Posted by Hari Shreedharan <hs...@cloudera.com>.
Running on Windows is something I have no idea about. Perhaps Roshan can help?


Thanks,
Hari

On Tue, Nov 11, 2014 at 9:08 PM, mahendran m <ma...@hotmail.com>
wrote:

> Hi Hari ,
> Thanks  for your reply .
> Q: How did you stop the agent?
> A: I was start agent from widows command prompt and simply closing the agent by clicking close button of command prompt
> Q: the application which you wrote - does it handle resending the events when Avro Source throws an exception. It looks like Avro Source received a bunch of events, then you killed the agent and did not resend?
> A: Yes  i am sending the event to source then you killed agent when sending event is in process .Can you please explain me how can i implement the re-sending event in my application 
> Thanks,Mahendran
> Date: Tue, 11 Nov 2014 11:50:21 -0800
> From: hshreedharan@cloudera.com
> To: user@flume.apache.org
> CC: user@flume.apache.org
> Subject: Re: File channel data lost
> the application which you wrote - does it handle resending the events when Avro Source throws an exception. It looks like Avro Source received a bunch of events, then you killed the agent and did not resend?
> #2 is expected. We don’t append to the file, instead we create a new file. There is nothing wrong with it. 
> How did you stop the agent? If you killed it with kill -9 then the rename will not happen.
> Thanks,
> Hari
> On Tue, Nov 11, 2014 at 2:22 AM, mahendran m <ma...@hotmail.com> wrote:
> Hi All ,
> I have just implemented the Flume agent with below configuration 
> Configuration
> # example.conf: A single-node Flume configuration
> # Name the components on this agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
> # Describe/configure the source
> a1.sources.r1.type = avro
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 44440
> # Describe the sink
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.fileType = DataStream
> a1.sinks.k1.hdfs.fileSuffix= .txt
> a1.sinks.k1.hdfs.rollSize = 1048576
> a1.sinks.k1.hdfs.rollCount = 0
> a1.sinks.k1.hdfs.rollInterval = 0
> a1.sinks.k1.hdfs.batchSize = 1000
> a1.sinks.k1.hdfs.minBlockReplicas = 1
> a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flume/MemoryChannel/Avro
>  
> #using the file channel
> a1.channels.c1.type = file
> a1.channels.c1.capacity = 1000000
> a1.channels.c1.transactionCapacity = 10000
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> Now i am sending batch of 1000 event to flume AVRO source and each event with UID incremented by one. HDFS server  create text file of 1MB each as per my configuration and file with .tmp extension (file which is process now). Now i stopping the flume agent and start it again. below are my two Expectation when starting flume agent again
> 1. Agent will resend the event from next to last successfully received event (in my case .tmp file has event with UID 12000 as     last so next event will be event with UID as 12001 )
> But what append is it start event with 12500 UID , event from 12001 to 12499 is completely lost
> 2. Agent will resume the appending event to file where it left last that is file which is not completed (file with .tmp extension)
> But agent not resumed the appending event to file where it had left . it created the new text file and start to append it .
> Can any one explain we why my two expectation failed ?
> And also file are remained with .tmp extension once i stopped the agent it doesn't remove this extension . can any know why these happening ? 
> Regards,
> Mahendran 		 	   		  
>  		 	   		  
>  		 	   		  

RE: File channel data lost

Posted by mahendran m <ma...@hotmail.com>.
Hi Hari ,
Thanks  for your reply .
Q: How did you stop the agent?
A: I was start agent from widows command prompt and simply closing the agent by clicking close button of command prompt

Q: the application which you wrote - does it handle resending the events when Avro Source throws an exception. It looks like Avro Source received a bunch of events, then you killed the agent and did not resend?
A: Yes  i am sending the event to source then you killed agent when sending event is in process .Can you please explain me how can i implement the re-sending event in my application 
Thanks,Mahendran
Date: Tue, 11 Nov 2014 11:50:21 -0800
From: hshreedharan@cloudera.com
To: user@flume.apache.org
CC: user@flume.apache.org
Subject: Re: File channel data lost


the application which you wrote - does it handle resending the events when Avro Source throws an exception. It looks like Avro Source received a bunch of events, then you killed the agent and did not resend?


#2 is expected. We don’t append to the file, instead we create a new file. There is nothing wrong with it. 


How did you stop the agent? If you killed it with kill -9 then the rename will not happen.

Thanks,
Hari


On Tue, Nov 11, 2014 at 2:22 AM, mahendran m <ma...@hotmail.com> wrote:




Hi All ,


I have just implemented the Flume agent with below configuration 


Configuration



# example.conf: A single-node Flume configuration


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44440


# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.fileSuffix= .txt
a1.sinks.k1.hdfs.rollSize = 1048576
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.batchSize = 1000
a1.sinks.k1.hdfs.minBlockReplicas = 1
a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flume/MemoryChannel/Avro
 
#using the file channel
a1.channels.c1.type = file
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1





Now i am sending batch of 1000 event to flume AVRO source and each event with UID incremented by one. HDFS server  create text file of 1MB each as per my configuration and file with .tmp extension (file which is process now). Now i stopping the flume agent and start it again. below are my two Expectation when starting flume agent again




1. Agent will resend the event from next to last successfully received event (in my case .tmp file has event with UID 12000 as     last so next event will be event with UID as 12001 )

But what append is it start event with 12500 UID , event from 12001 to 12499 is completely lost



2. Agent will resume the appending event to file where it left last that is file which is not completed (file with .tmp extension)
But agent not resumed the appending event to file where it had left . it created the new text file and start to append it .





Can any one explain we why my two expectation failed ?


And also file are remained with .tmp extension once i stopped the agent it doesn't remove this extension . can any know why these happening ? 


Regards,
Mahendran 		 	   		  

 		 	   		  
 		 	   		  

Re: File channel data lost

Posted by Hari Shreedharan <hs...@cloudera.com>.
the application which you wrote - does it handle resending the events when Avro Source throws an exception. It looks like Avro Source received a bunch of events, then you killed the agent and did not resend?




#2 is expected. We don’t append to the file, instead we create a new file. There is nothing wrong with it. 




How did you stop the agent? If you killed it with kill -9 then the rename will not happen.


Thanks,
Hari

On Tue, Nov 11, 2014 at 2:22 AM, mahendran m <ma...@hotmail.com>
wrote:

> Hi All ,
> I have just implemented the Flume agent with below configuration 
> Configuration
> # example.conf: A single-node Flume configuration
> # Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1
> # Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.bind = localhosta1.sources.r1.port = 44440
> # Describe the sinka1.sinks.k1.type = hdfsa1.sinks.k1.hdfs.fileType = DataStreama1.sinks.k1.hdfs.fileSuffix= .txta1.sinks.k1.hdfs.rollSize = 1048576a1.sinks.k1.hdfs.rollCount = 0a1.sinks.k1.hdfs.rollInterval = 0a1.sinks.k1.hdfs.batchSize = 1000a1.sinks.k1.hdfs.minBlockReplicas = 1a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flume/MemoryChannel/Avro #using the file channela1.channels.c1.type = filea1.channels.c1.capacity = 1000000a1.channels.c1.transactionCapacity = 10000
> # Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1
> Now i am sending batch of 1000 event to flume AVRO source and each event with UID incremented by one. HDFS server  create text file of 1MB each as per my configuration and file with .tmp extension (file which is process now). Now i stopping the flume agent and start it again. below are my two Expectation when starting flume agent again
> 1. Agent will resend the event from next to last successfully received event (in my case .tmp file has event with UID 12000 as     last so next event will be event with UID as 12001 )But what append is it start event with 12500 UID , event from 12001 to 12499 is completely lost
> 2. Agent will resume the appending event to file where it left last that is file which is not completed (file with .tmp extension)But agent not resumed the appending event to file where it had left . it created the new text file and start to append it .
> Can any one explain we why my two expectation failed ?
> And also file are remained with .tmp extension once i stopped the agent it doesn't remove this extension . can any know why these happening ? 
> Regards,Mahendran 		 	   		  
>  		 	   		  

File channel data lost

Posted by mahendran m <ma...@hotmail.com>.


Hi All ,
I have just implemented the Flume agent with below configuration 
Configuration
# example.conf: A single-node Flume configuration
# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1
# Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.bind = localhosta1.sources.r1.port = 44440
# Describe the sinka1.sinks.k1.type = hdfsa1.sinks.k1.hdfs.fileType = DataStreama1.sinks.k1.hdfs.fileSuffix= .txta1.sinks.k1.hdfs.rollSize = 1048576a1.sinks.k1.hdfs.rollCount = 0a1.sinks.k1.hdfs.rollInterval = 0a1.sinks.k1.hdfs.batchSize = 1000a1.sinks.k1.hdfs.minBlockReplicas = 1a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flume/MemoryChannel/Avro #using the file channela1.channels.c1.type = filea1.channels.c1.capacity = 1000000a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

Now i am sending batch of 1000 event to flume AVRO source and each event with UID incremented by one. HDFS server  create text file of 1MB each as per my configuration and file with .tmp extension (file which is process now). Now i stopping the flume agent and start it again. below are my two Expectation when starting flume agent again
1. Agent will resend the event from next to last successfully received event (in my case .tmp file has event with UID 12000 as     last so next event will be event with UID as 12001 )But what append is it start event with 12500 UID , event from 12001 to 12499 is completely lost
2. Agent will resume the appending event to file where it left last that is file which is not completed (file with .tmp extension)But agent not resumed the appending event to file where it had left . it created the new text file and start to append it .
Can any one explain we why my two expectation failed ?
And also file are remained with .tmp extension once i stopped the agent it doesn't remove this extension . can any know why these happening ? 
Regards,Mahendran 		 	   		  
 		 	   		  

RE: Perfromacnce issue in file channel

Posted by mahendran m <ma...@hotmail.com>.
Hi Hari ,
Thanks , yes i am sending the event one by one only as it is my requirement but memory channel can easily process 1000 per second but file channel doesn't is this case how can i improve the performance of file channel .
Regards,Mahendran

Date: Mon, 10 Nov 2014 11:34:57 -0800
From: hshreedharan@cloudera.com
To: user@flume.apache.org
CC: user@flume.apache.org
Subject: Re: Perfromacnce issue in file channel


Sending one event by one event is likely to cause performance issues. In addition, I have no idea of the performance on Windows. Btw, 100-200 request per second is really orders of magnitude lower than what File Channel can handle.

Thanks,
Hari


On Sun, Nov 9, 2014 at 7:49 AM, mahendran m <ma...@hotmail.com> wrote:




Hi All ,
I have tested the file channel and file channel performance. below the code snippet to generate 1000 event per sec 




while (watch.Elapsed.TotalSeconds < sec)
{

	if (counter <= 1000)

	{

		uuid = uuid + 1;



		var text = HttpUtility.UrlEncode("HI , How are you", Encoding.UTF8);



		var flumeEvent = new ThriftFlumeEvent();

		flumeEvent.Headers = new Dictionary<string, string>() { { "Host", "http://www.bing.com/" } };

		flumeEvent.Body = Encoding.ASCII.GetBytes(text);

		Client.append(flumeEvent);

		counter += 1;

	}
}



i succeed in sending the 1000 event per sec when using memory channel , but in case if i used file channel  i can  only reach upto 40 to 50 events per second


config for memory channel

# Use memory channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 10
a1.channels.c1.byteCapacity = 5368709120





config for file channel



#using the file channel
a1.channels.c1.type = file
a1.channels.c1.capacity = 500000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = F:/.checkPoint


Can anyone explain why file channel is this much channel ? . if i used file channel in server for directly logging each request is it better option as my server receive 100 to 200 request per second


Regards,
Mahendran




 		 	   		  

 		 	   		  

Re: Perfromacnce issue in file channel

Posted by Hari Shreedharan <hs...@cloudera.com>.
Sending one event by one event is likely to cause performance issues. In addition, I have no idea of the performance on Windows. Btw, 100-200 request per second is really orders of magnitude lower than what File Channel can handle.


Thanks,
Hari

On Sun, Nov 9, 2014 at 7:49 AM, mahendran m <ma...@hotmail.com>
wrote:

> Hi All ,I have tested the file channel and file channel performance. below the code snippet to generate 1000 event per sec 
> while (watch.Elapsed.TotalSeconds < sec){	if (counter <= 1000)	{		uuid = uuid + 1;
> 		var text = HttpUtility.UrlEncode("HI , How are you", Encoding.UTF8);
> 		var flumeEvent = new ThriftFlumeEvent();		flumeEvent.Headers = new Dictionary<string, string>() { { "Host", "http://www.bing.com/" } };		flumeEvent.Body = Encoding.ASCII.GetBytes(text);		Client.append(flumeEvent);		counter += 1;	}}
> i succeed in sending the 1000 event per sec when using memory channel , but in case if i used file channel  i can  only reach upto 40 to 50 events per second
> config for memory channel# Use memory channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000000a1.channels.c1.transactionCapacity = 1000a1.channels.c1.byteCapacityBufferPercentage = 10a1.channels.c1.byteCapacity = 5368709120
> config for file channel
> #using the file channela1.channels.c1.type = filea1.channels.c1.capacity = 500000a1.channels.c1.transactionCapacity = 10000a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = F:/.checkPoint
> Can anyone explain why file channel is this much channel ? . if i used file channel in server for directly logging each request is it better option as my server receive 100 to 200 request per second
> Regards,Mahendran
>