You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Shangan Chen <ch...@gmail.com> on 2014/01/21 10:38:57 UTC
flume use quite a lot of direct momery and blast
I have quite a lot of flume-agents streaming logs to several
flume-collectors. The problem I face is the memory consumed by
flume-collector is increasing all the time but slowly even I limit the max
heap. I know flume use NIO, but I don't know why it cause memory increase
boundlessly. I prints some information I can get, I really appreciate if
anyone can give any help.
*my jvm conf:*
JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
*top prints of flume instance:*
top - 17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86,
0.79
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si,
0.4%st
Mem: 16435540k total, 16194572k used, 240968k free, 25896k buffers
Swap: 8385892k total, 205592k used, 8180300k free, 165968k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 java
*jmap prints :*
jmap -heap 16580
Attaching to process ID 16580, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.21-b01
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 8589934592 (8192.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 2105147392 (2007.625MB)
used = 1048958560 (1000.3648376464844MB)
free = 1056188832 (1007.2601623535156MB)
49.828271596861185% used
>From Space:
capacity = 369229824 (352.125MB)
used = 369168400 (352.06642150878906MB)
free = 61424 (0.0585784912109375MB)
99.98336429074591% used
To Space:
capacity = 388890624 (370.875MB)
used = 0 (0.0MB)
free = 388890624 (370.875MB)
0.0% used
PS Old Generation
capacity = 5726666752 (5461.375MB)
used = 4282458544 (4084.0707244873047MB)
free = 1444208208 (1377.3042755126953MB)
74.78099790780352% used
PS Perm Generation
capacity = 37879808 (36.125MB)
used = 37833736 (36.08106231689453MB)
free = 46072 (0.04393768310546875MB)
99.8783731955558% used
11441 interned Strings occupying 1025920 bytes.
*DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
<https://gist.github.com/rednaxelafx/1593521>) :*
java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
Attaching to process ID 16580, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.21-b01
NIO direct memory: (in bytes)
reserved size = 5864.191787 MB (6149050767 bytes)
max size = 7850.687500 MB (8232042496 bytes)
--
have a good day!
chenshang'an
Re: flume use quite a lot of direct momery and blast
Posted by Brock Noland <br...@cloudera.com>.
Are you using hdfs sink? If so please share your configuration.
I have seen this when the sink keeps many open files...each file holding on
to a native compression buffer.
On Jan 22, 2014 1:37 AM, "Shangan Chen" <ch...@gmail.com> wrote:
> Thanks a lot for your reply. I deployed flume on ubuntu and what confused
> me is not about the virtual address space but NIO Direct Memory increasing
> boundlessly. It will finally cause the machine to report swap is low.
>
>
> On Wed, Jan 22, 2014 at 1:08 AM, Brock Noland <br...@cloudera.com> wrote:
>
>> Looks like it's arena allocation. Basically it's nothing to worry about
>> since virtual address space on 64bit machines is not in short supply. You
>> could limit it with a parameter like so:
>>
>>
>> https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
>>
>>
>> On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <ch...@gmail.com>wrote:
>>
>>> I have quite a lot of flume-agents streaming logs to several
>>> flume-collectors. The problem I face is the memory consumed by
>>> flume-collector is increasing all the time but slowly even I limit the max
>>> heap. I know flume use NIO, but I don't know why it cause memory increase
>>> boundlessly. I prints some information I can get, I really appreciate if
>>> anyone can give any help.
>>>
>>> *my jvm conf:*
>>>
>>> JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
>>>
>>> *top prints of flume instance:*
>>>
>>> top - 17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86,
>>> 0.79
>>> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
>>> Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si,
>>> 0.4%st
>>> Mem: 16435540k total, 16194572k used, 240968k free, 25896k buffers
>>> Swap: 8385892k total, 205592k used, 8180300k free, 165968k cached
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 java
>>>
>>>
>>> *jmap prints :*
>>>
>>> jmap -heap 16580
>>> Attaching to process ID 16580, please wait...
>>> Debugger attached successfully.
>>> Server compiler detected.
>>> JVM version is 23.21-b01
>>>
>>> using thread-local object allocation.
>>> Parallel GC with 8 thread(s)
>>>
>>> Heap Configuration:
>>> MinHeapFreeRatio = 40
>>> MaxHeapFreeRatio = 70
>>> MaxHeapSize = 8589934592 (8192.0MB)
>>> NewSize = 1310720 (1.25MB)
>>> MaxNewSize = 17592186044415 MB
>>> OldSize = 5439488 (5.1875MB)
>>> NewRatio = 2
>>> SurvivorRatio = 8
>>> PermSize = 21757952 (20.75MB)
>>> MaxPermSize = 85983232 (82.0MB)
>>> G1HeapRegionSize = 0 (0.0MB)
>>>
>>> Heap Usage:
>>> PS Young Generation
>>> Eden Space:
>>> capacity = 2105147392 (2007.625MB)
>>> used = 1048958560 (1000.3648376464844MB)
>>> free = 1056188832 (1007.2601623535156MB)
>>> 49.828271596861185% used
>>> From Space:
>>> capacity = 369229824 (352.125MB)
>>> used = 369168400 (352.06642150878906MB)
>>> free = 61424 (0.0585784912109375MB)
>>> 99.98336429074591% used
>>> To Space:
>>> capacity = 388890624 (370.875MB)
>>> used = 0 (0.0MB)
>>> free = 388890624 (370.875MB)
>>> 0.0% used
>>> PS Old Generation
>>> capacity = 5726666752 (5461.375MB)
>>> used = 4282458544 (4084.0707244873047MB)
>>> free = 1444208208 (1377.3042755126953MB)
>>> 74.78099790780352% used
>>> PS Perm Generation
>>> capacity = 37879808 (36.125MB)
>>> used = 37833736 (36.08106231689453MB)
>>> free = 46072 (0.04393768310546875MB)
>>> 99.8783731955558% used
>>>
>>> 11441 interned Strings occupying 1025920 bytes.
>>>
>>> *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
>>> <https://gist.github.com/rednaxelafx/1593521>) :*
>>>
>>> java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
>>> Attaching to process ID 16580, please wait...
>>> Debugger attached successfully.
>>> Server compiler detected.
>>> JVM version is 23.21-b01
>>> NIO direct memory: (in bytes)
>>> reserved size = 5864.191787 MB (6149050767 bytes)
>>> max size = 7850.687500 MB (8232042496 bytes)
>>>
>>>
>>>
>>> --
>>> have a good day!
>>> chenshang'an
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>
>
> --
> have a good day!
> chenshang'an
>
>
Re: flume use quite a lot of direct momery and blast
Posted by Shangan Chen <ch...@gmail.com>.
Thanks a lot for your reply. I deployed flume on ubuntu and what confused
me is not about the virtual address space but NIO Direct Memory increasing
boundlessly. It will finally cause the machine to report swap is low.
On Wed, Jan 22, 2014 at 1:08 AM, Brock Noland <br...@cloudera.com> wrote:
> Looks like it's arena allocation. Basically it's nothing to worry about
> since virtual address space on 64bit machines is not in short supply. You
> could limit it with a parameter like so:
>
>
> https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
>
>
> On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <ch...@gmail.com>wrote:
>
>> I have quite a lot of flume-agents streaming logs to several
>> flume-collectors. The problem I face is the memory consumed by
>> flume-collector is increasing all the time but slowly even I limit the max
>> heap. I know flume use NIO, but I don't know why it cause memory increase
>> boundlessly. I prints some information I can get, I really appreciate if
>> anyone can give any help.
>>
>> *my jvm conf:*
>>
>> JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
>>
>> *top prints of flume instance:*
>>
>> top - 17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86,
>> 0.79
>> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si,
>> 0.4%st
>> Mem: 16435540k total, 16194572k used, 240968k free, 25896k buffers
>> Swap: 8385892k total, 205592k used, 8180300k free, 165968k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 java
>>
>>
>> *jmap prints :*
>>
>> jmap -heap 16580
>> Attaching to process ID 16580, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 23.21-b01
>>
>> using thread-local object allocation.
>> Parallel GC with 8 thread(s)
>>
>> Heap Configuration:
>> MinHeapFreeRatio = 40
>> MaxHeapFreeRatio = 70
>> MaxHeapSize = 8589934592 (8192.0MB)
>> NewSize = 1310720 (1.25MB)
>> MaxNewSize = 17592186044415 MB
>> OldSize = 5439488 (5.1875MB)
>> NewRatio = 2
>> SurvivorRatio = 8
>> PermSize = 21757952 (20.75MB)
>> MaxPermSize = 85983232 (82.0MB)
>> G1HeapRegionSize = 0 (0.0MB)
>>
>> Heap Usage:
>> PS Young Generation
>> Eden Space:
>> capacity = 2105147392 (2007.625MB)
>> used = 1048958560 (1000.3648376464844MB)
>> free = 1056188832 (1007.2601623535156MB)
>> 49.828271596861185% used
>> From Space:
>> capacity = 369229824 (352.125MB)
>> used = 369168400 (352.06642150878906MB)
>> free = 61424 (0.0585784912109375MB)
>> 99.98336429074591% used
>> To Space:
>> capacity = 388890624 (370.875MB)
>> used = 0 (0.0MB)
>> free = 388890624 (370.875MB)
>> 0.0% used
>> PS Old Generation
>> capacity = 5726666752 (5461.375MB)
>> used = 4282458544 (4084.0707244873047MB)
>> free = 1444208208 (1377.3042755126953MB)
>> 74.78099790780352% used
>> PS Perm Generation
>> capacity = 37879808 (36.125MB)
>> used = 37833736 (36.08106231689453MB)
>> free = 46072 (0.04393768310546875MB)
>> 99.8783731955558% used
>>
>> 11441 interned Strings occupying 1025920 bytes.
>>
>> *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
>> <https://gist.github.com/rednaxelafx/1593521>) :*
>>
>> java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
>> Attaching to process ID 16580, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 23.21-b01
>> NIO direct memory: (in bytes)
>> reserved size = 5864.191787 MB (6149050767 bytes)
>> max size = 7850.687500 MB (8232042496 bytes)
>>
>>
>>
>> --
>> have a good day!
>> chenshang'an
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
have a good day!
chenshang'an
Re: flume use quite a lot of direct momery and blast
Posted by Brock Noland <br...@cloudera.com>.
Looks like it's arena allocation. Basically it's nothing to worry about
since virtual address space on 64bit machines is not in short supply. You
could limit it with a parameter like so:
https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <ch...@gmail.com>wrote:
> I have quite a lot of flume-agents streaming logs to several
> flume-collectors. The problem I face is the memory consumed by
> flume-collector is increasing all the time but slowly even I limit the max
> heap. I know flume use NIO, but I don't know why it cause memory increase
> boundlessly. I prints some information I can get, I really appreciate if
> anyone can give any help.
>
> *my jvm conf:*
>
> JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
>
> *top prints of flume instance:*
>
> top - 17:28:59 up 103 days, 34 min, 4 users, load average: 0.82, 0.86,
> 0.79
> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
> Cpu(s): 5.4%us, 4.7%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.1%si,
> 0.4%st
> Mem: 16435540k total, 16194572k used, 240968k free, 25896k buffers
> Swap: 8385892k total, 205592k used, 8180300k free, 165968k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 16580 sankuai 20 0 23.1g 14g 5464 S 92 93.7 7522:53 java
>
>
> *jmap prints :*
>
> jmap -heap 16580
> Attaching to process ID 16580, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 23.21-b01
>
> using thread-local object allocation.
> Parallel GC with 8 thread(s)
>
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 8589934592 (8192.0MB)
> NewSize = 1310720 (1.25MB)
> MaxNewSize = 17592186044415 MB
> OldSize = 5439488 (5.1875MB)
> NewRatio = 2
> SurvivorRatio = 8
> PermSize = 21757952 (20.75MB)
> MaxPermSize = 85983232 (82.0MB)
> G1HeapRegionSize = 0 (0.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
> capacity = 2105147392 (2007.625MB)
> used = 1048958560 (1000.3648376464844MB)
> free = 1056188832 (1007.2601623535156MB)
> 49.828271596861185% used
> From Space:
> capacity = 369229824 (352.125MB)
> used = 369168400 (352.06642150878906MB)
> free = 61424 (0.0585784912109375MB)
> 99.98336429074591% used
> To Space:
> capacity = 388890624 (370.875MB)
> used = 0 (0.0MB)
> free = 388890624 (370.875MB)
> 0.0% used
> PS Old Generation
> capacity = 5726666752 (5461.375MB)
> used = 4282458544 (4084.0707244873047MB)
> free = 1444208208 (1377.3042755126953MB)
> 74.78099790780352% used
> PS Perm Generation
> capacity = 37879808 (36.125MB)
> used = 37833736 (36.08106231689453MB)
> free = 46072 (0.04393768310546875MB)
> 99.8783731955558% used
>
> 11441 interned Strings occupying 1025920 bytes.
>
> *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
> <https://gist.github.com/rednaxelafx/1593521>) :*
>
> java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
> Attaching to process ID 16580, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 23.21-b01
> NIO direct memory: (in bytes)
> reserved size = 5864.191787 MB (6149050767 bytes)
> max size = 7850.687500 MB (8232042496 bytes)
>
>
>
> --
> have a good day!
> chenshang'an
>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org