You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Kumar, Deepak8 " <de...@citi.com> on 2013/01/17 07:34:22 UTC
FW: Flume Service Takes Lot of Memory
Hi,
Could you please guide?
Regards,
Deepak
-----Original Message-----
From: Kumar, Deepak8 [CCC-OT_IT NE]
Sent: Wednesday, January 16, 2013 1:29 PM
To: cdh-user@cloudera.org
Subject: Flume Service Takes Lot of Memory
Hi,
I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
Regards,
Deepak
Re: FW: Flume Service Takes Lot of Memory
Posted by Nitin Pawar <ni...@gmail.com>.
can you share your conf file
On Thu, Jan 17, 2013 at 12:04 PM, Kumar, Deepak8 <de...@citi.com>wrote:
> Hi,
> Could you please guide?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Kumar, Deepak8 [CCC-OT_IT NE]
> Sent: Wednesday, January 16, 2013 1:29 PM
> To: cdh-user@cloudera.org
> Subject: Flume Service Takes Lot of Memory
>
> Hi,
> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual
> Memory. Is it uses this much normally or do we have some configuration
> settings?
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java
> -Dflume.log.dir=/var/log/flume 4957 root
>
> Regards,
> Deepak
>
--
Nitin Pawar
Re: Flume Service Takes Lot of Memory
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
No, this will work for the master only. To pass this to agents you have to tweak the startup script after line 104 ($FLUME_HOME/bin/flume):
https://github.com/cloudera/flume/blob/master/bin/flume#L104
JOPTS="$JOPTS -Xms1g -Xmx2g "
as example.
- Alex
On Jan 17, 2013, at 10:40 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
> Hi Alex,
> The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG & MASTERI_WATCHDOG at flume master will work for agent memory as well? Or do we have some other option as flume agent?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> Sent: Thursday, January 17, 2013 2:18 AM
> To: user@flume.apache.org
> Subject: Re: Flume Service Takes Lot of Memory
>
> Hi,
>
> To add JVM options in flume we have to edit the watchdog line in bin/flume into:
> MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
> MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
>
> and restart Flume master.
>
> flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and FLUME_IDENT_STRING
>
> Ensure that you have provided the jvm with sufficient heap space. By default Flume starts the jvm with it's default heap allocation, which differs depending on the jvm version, the host type (os, 32/64 bit, etc...), total host memory available, as well as other issues.
>
> The environment variable UOPTS can be used to pass additional jvm parameters when running Flume. e.g.
>
> $ UOPTS="-Xms1g -Xmx2g" bin/flume node
>
> which starts a flume node with an initial heap of one gig and a max heap of two gig. See "java -h" or "java -X" for more details on available jvm options.
>
> cheers,
> Alex
>
> On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
>
>> Hi Alex,
>> Thanks for assisting. I do have the following settings in flume-env.sh
>> file export UOPTS="-Xms256m -Xmx512m"
>>
>> But once I do top, it shows this much virtual memory
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>>
>> Regards,
>> Deepak
>>
>> -----Original Message-----
>> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
>> Sent: Thursday, January 17, 2013 1:58 AM
>> To: user@flume.apache.org
>> Subject: Re: Flume Service Takes Lot of Memory
>>
>> Hi Deepak,
>>
>> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
>>
>> UOPTS="-Xms1g -Xmx2g" bin/flume node
>>
>> Of course you can also tweak the bin/flume script.
>>
>> - Alex
>>
>> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
>>
>>> Hi,
>>> Could you please guide?
>>>
>>> Regards,
>>> Deepak
>>>
>>> -----Original Message-----
>>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>>> Sent: Wednesday, January 16, 2013 1:29 PM
>>> To: cdh-user@cloudera.org
>>> Subject: Flume Service Takes Lot of Memory
>>>
>>> Hi,
>>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>>>
>>> Regards,
>>> Deepak
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Flume Service Takes Lot of Memory
Posted by "Kumar, Deepak8 " <de...@citi.com>.
Hi Alex,
The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG & MASTERI_WATCHDOG at flume master will work for agent memory as well? Or do we have some other option as flume agent?
Regards,
Deepak
-----Original Message-----
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Thursday, January 17, 2013 2:18 AM
To: user@flume.apache.org
Subject: Re: Flume Service Takes Lot of Memory
Hi,
To add JVM options in flume we have to edit the watchdog line in bin/flume into:
MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
and restart Flume master.
flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and FLUME_IDENT_STRING
Ensure that you have provided the jvm with sufficient heap space. By default Flume starts the jvm with it's default heap allocation, which differs depending on the jvm version, the host type (os, 32/64 bit, etc...), total host memory available, as well as other issues.
The environment variable UOPTS can be used to pass additional jvm parameters when running Flume. e.g.
$ UOPTS="-Xms1g -Xmx2g" bin/flume node
which starts a flume node with an initial heap of one gig and a max heap of two gig. See "java -h" or "java -X" for more details on available jvm options.
cheers,
Alex
On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
> Hi Alex,
> Thanks for assisting. I do have the following settings in flume-env.sh
> file export UOPTS="-Xms256m -Xmx512m"
>
> But once I do top, it shows this much virtual memory
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> Sent: Thursday, January 17, 2013 1:58 AM
> To: user@flume.apache.org
> Subject: Re: Flume Service Takes Lot of Memory
>
> Hi Deepak,
>
> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
>
> UOPTS="-Xms1g -Xmx2g" bin/flume node
>
> Of course you can also tweak the bin/flume script.
>
> - Alex
>
> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
>
>> Hi,
>> Could you please guide?
>>
>> Regards,
>> Deepak
>>
>> -----Original Message-----
>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>> Sent: Wednesday, January 16, 2013 1:29 PM
>> To: cdh-user@cloudera.org
>> Subject: Flume Service Takes Lot of Memory
>>
>> Hi,
>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>>
>> Regards,
>> Deepak
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Flume Service Takes Lot of Memory
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi,
To add JVM options in flume we have to edit the watchdog line in bin/flume into:
MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
and restart Flume master.
flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and FLUME_IDENT_STRING
Ensure that you have provided the jvm with sufficient heap space. By default Flume starts the jvm with it’s default heap allocation, which differs depending on the jvm version, the host type (os, 32/64 bit, etc…), total host memory available, as well as other issues.
The environment variable UOPTS can be used to pass additional jvm parameters when running Flume. e.g.
$ UOPTS=“-Xms1g -Xmx2g” bin/flume node
which starts a flume node with an initial heap of one gig and a max heap of two gig. See “java -h” or “java -X” for more details on available jvm options.
cheers,
Alex
On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
> Hi Alex,
> Thanks for assisting. I do have the following settings in flume-env.sh file
> export UOPTS="-Xms256m -Xmx512m"
>
> But once I do top, it shows this much virtual memory
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> Sent: Thursday, January 17, 2013 1:58 AM
> To: user@flume.apache.org
> Subject: Re: Flume Service Takes Lot of Memory
>
> Hi Deepak,
>
> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
>
> UOPTS="-Xms1g -Xmx2g" bin/flume node
>
> Of course you can also tweak the bin/flume script.
>
> - Alex
>
> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
>
>> Hi,
>> Could you please guide?
>>
>> Regards,
>> Deepak
>>
>> -----Original Message-----
>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>> Sent: Wednesday, January 16, 2013 1:29 PM
>> To: cdh-user@cloudera.org
>> Subject: Flume Service Takes Lot of Memory
>>
>> Hi,
>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>>
>> Regards,
>> Deepak
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: Flume Service Takes Lot of Memory
Posted by "Kumar, Deepak8 " <de...@citi.com>.
Hi Alex,
Thanks for assisting. I do have the following settings in flume-env.sh file
export UOPTS="-Xms256m -Xmx512m"
But once I do top, it shows this much virtual memory
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
Regards,
Deepak
-----Original Message-----
From: Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
Sent: Thursday, January 17, 2013 1:58 AM
To: user@flume.apache.org
Subject: Re: Flume Service Takes Lot of Memory
Hi Deepak,
This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
UOPTS="-Xms1g -Xmx2g" bin/flume node
Of course you can also tweak the bin/flume script.
- Alex
On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
> Hi,
> Could you please guide?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Kumar, Deepak8 [CCC-OT_IT NE]
> Sent: Wednesday, January 16, 2013 1:29 PM
> To: cdh-user@cloudera.org
> Subject: Flume Service Takes Lot of Memory
>
> Hi,
> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>
> Regards,
> Deepak
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Flume Service Takes Lot of Memory
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Hi Deepak,
This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
UOPTS=“-Xms1g -Xmx2g” bin/flume node
Of course you can also tweak the bin/flume script.
- Alex
On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <de...@citi.com> wrote:
> Hi,
> Could you please guide?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Kumar, Deepak8 [CCC-OT_IT NE]
> Sent: Wednesday, January 16, 2013 1:29 PM
> To: cdh-user@cloudera.org
> Subject: Flume Service Takes Lot of Memory
>
> Hi,
> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java -Dflume.log.dir=/var/log/flume 4957 root
>
> Regards,
> Deepak
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF