You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by MrWell <hu...@foxmail.com> on 2019/11/22 07:00:47 UTC

metastore clean OutOfMemoryError

Hi, Kylin Team.

When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this



java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4839.hprof ...
Heap dump file created [317991670&nbsp;bytes in 2.120 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#&nbsp; &nbsp;Executing /bin/sh -c "kill -9 4839"...
bin/metastore.sh: line 109:&nbsp; 4839 Killed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"





I have set 'setenv.sh' file, like this


export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"


Dose it means heap memory is still small?

Re: metastore clean OutOfMemoryError

Posted by ShaoFeng Shi <sh...@apache.org>.
Excellent Marc!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




Marc Wu -X (mawu2 - Insigma Hengtian at Cisco) <ma...@cisco.com>
于2019年11月22日周五 下午3:16写道：

> Hi MrWell,
>
>
>
> You can try to create a file named setenv-tool.sh in the $KYLIN_HOME/conf
> directory, and put the following content in the file, then execute your
> command. It may solve your issues.
>
> #!/bin/bash
>
> #
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
>
> # contributor license agreements.  See the NOTICE file distributed with
>
> # this work for additional information regarding copyright ownership.
>
> # The ASF licenses this file to You under the Apache License, Version 2.0
>
> # (the "License"); you may not use this file except in compliance with
>
> # the License.  You may obtain a copy of the License at
>
> #
>
> #    http://www.apache.org/licenses/LICENSE-2.0
>
> #
>
> # Unless required by applicable law or agreed to in writing, software
>
> # distributed under the License is distributed on an "AS IS" BASIS,
>
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>
> # See the License for the specific language governing permissions and
>
> # limitations under the License.
>
> #
>
> export KYLIN_EXTRA_START_OPTS="-Xmx3072M"
>
>
>
>
>
> *From: *MrWell <hu...@foxmail.com>
> *Reply-To: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *Date: *Friday, November 22, 2019 at 15:01
> *To: *user <us...@kylin.apache.org>
> *Subject: *metastore clean OutOfMemoryError
>
>
>
> Hi, Kylin Team.
>
>
>
> When I execute "bin/metastore.sh clean --delete true" , I get a
> "OutOfMemoryError" like this
>
>
>
>
>
> java.lang.OutOfMemoryError: Java heap space
>
> Dumping heap to java_pid4839.hprof ...
>
> Heap dump file created [317991670 bytes in 2.120 secs]
>
> #
>
> # java.lang.OutOfMemoryError: Java heap space
>
> # -XX:OnOutOfMemoryError="kill -9 %p"
>
> #   Executing /bin/sh -c "kill -9 4839"...
>
> bin/metastore.sh: line 109:  4839 Killed
> ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
>
>
>
>
>
> I have set 'setenv.sh' file, like this
>
>
>
> export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m
> -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4
> -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
>
>
>
> Dose it means heap memory is still small?
>

回复： 回复： metastore clean OutOfMemoryError

Posted by MrWell <hu...@foxmail.com>.
Thanks!!!&nbsp; it dose work!&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"Marc Wu -X (mawu2 - Insigma Hengtian at Cisco)"<mawu2@cisco.com&gt;;
发送时间:&nbsp;2019年11月22日(星期五) 下午4:40
收件人:&nbsp;"user@kylin.apache.org"<user@kylin.apache.org&gt;;

主题:&nbsp;Re: 回复： metastore clean OutOfMemoryError



  
Sorry for my mistake, can you move the setenv-tool.sh to the $KYLIN_HOME/bin and try again?
 
&nbsp;
 
&nbsp;
  
From: MrWell <huangpeng_scut@foxmail.com&gt;
 Reply-To: "user@kylin.apache.org" <user@kylin.apache.org&gt;
 Date: Friday, November 22, 2019 at 16:24
 To: user <user@kylin.apache.org&gt;
 Subject: 回复： metastore clean OutOfMemoryError
 
  
&nbsp;
 
  
Hi Mrac Wu,
 
  
&nbsp;
 
  
Thanks, I have tried it, but it failed. 
 
   
&nbsp;
 
  
&nbsp;
 
  
------------------ 原始邮件&nbsp;------------------
 
   
发件人:&nbsp;"Marc Wu -X (mawu2 - Insigma Hengtian  at Cisco)"<mawu2@cisco.com&gt;;
 
  
发送时间:&nbsp;2019年11月22日(星期五) 下午3:16
 
  
收件人:&nbsp;"user@kylin.apache.org"<user@kylin.apache.org&gt;;
 
  
主题:&nbsp;Re: metastore clean OutOfMemoryError
 
 
  
&nbsp;
 
  
Hi MrWell, 
 
&nbsp;
 
You can try to create a file named setenv-tool.sh in the $KYLIN_HOME/conf directory, and put the following content in the file, then execute your command. It may solve your issues.
 
#!/bin/bash
 
#
 
# Licensed to the Apache Software Foundation (ASF) under one or more
 
# contributor license agreements.&nbsp; See the NOTICE file distributed with
 
# this work for additional information regarding copyright ownership.
 
# The ASF licenses this file to You under the Apache License, Version 2.0
 
# (the "License"); you may not use this file except in compliance with
 
# the License.&nbsp; You may obtain a copy of the License at
 
#
 
#&nbsp; &nbsp; http://www.apache.org/licenses/LICENSE-2.0
 
#
 
# Unless required by applicable law or agreed to in writing, software
 
# distributed under the License is distributed on an "AS IS" BASIS,
 
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 
# See the License for the specific language governing permissions and
 
# limitations under the License.
 
# 
 
export KYLIN_EXTRA_START_OPTS="-Xmx3072M"
 
&nbsp;
 
&nbsp;
  
From: MrWell <huangpeng_scut@foxmail.com&gt;
 Reply-To: "user@kylin.apache.org" <user@kylin.apache.org&gt;
 Date: Friday, November 22, 2019 at 15:01
 To: user <user@kylin.apache.org&gt;
 Subject: metastore clean OutOfMemoryError
 
  
&nbsp;
 
  
Hi, Kylin Team. 
  
&nbsp;
 
  
When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this
 
  
&nbsp;
 
 
&nbsp;
   
java.lang.OutOfMemoryError: Java heap space
 
  
Dumping heap to java_pid4839.hprof ...
 
  
Heap dump file created [317991670 bytes in 2.120 secs]
 
  
#
 
  
# java.lang.OutOfMemoryError: Java heap space
 
  
# -XX:OnOutOfMemoryError="kill -9 %p"
 
  
#&nbsp; &nbsp;Executing /bin/sh -c "kill -9 4839"...
 
  
bin/metastore.sh: line 109:&nbsp; 4839 Killed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
 
 
  
&nbsp;
 
  
&nbsp;
 
  
I have set 'setenv.sh' file, like this
 
  
&nbsp;
 
  
export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$  -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
 
  
&nbsp;
 
  
Dose it means heap memory is still small?

Re: 回复： metastore clean OutOfMemoryError

Posted by "Marc Wu -X (mawu2 - Insigma Hengtian at Cisco)" <ma...@cisco.com>.
Sorry for my mistake, can you move the setenv-tool.sh to the $KYLIN_HOME/bin and try again?


From: MrWell <hu...@foxmail.com>
Reply-To: "user@kylin.apache.org" <us...@kylin.apache.org>
Date: Friday, November 22, 2019 at 16:24
To: user <us...@kylin.apache.org>
Subject: 回复： metastore clean OutOfMemoryError

Hi Mrac Wu,

Thanks, I have tried it, but it failed.


------------------ 原始邮件 ------------------
发件人: "Marc Wu -X (mawu2 - Insigma Hengtian at Cisco)"<ma...@cisco.com>;
发送时间: 2019年11月22日(星期五) 下午3:16
收件人: "user@kylin.apache.org"<us...@kylin.apache.org>;
主题: Re: metastore clean OutOfMemoryError

Hi MrWell,

You can try to create a file named setenv-tool.sh in the $KYLIN_HOME/conf directory, and put the following content in the file, then execute your command. It may solve your issues.

#!/bin/bash

#

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

#

export KYLIN_EXTRA_START_OPTS="-Xmx3072M"


From: MrWell <hu...@foxmail.com>
Reply-To: "user@kylin.apache.org" <us...@kylin.apache.org>
Date: Friday, November 22, 2019 at 15:01
To: user <us...@kylin.apache.org>
Subject: metastore clean OutOfMemoryError

Hi, Kylin Team.

When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this


java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4839.hprof ...
Heap dump file created [317991670 bytes in 2.120 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 4839"...
bin/metastore.sh: line 109:  4839 Killed                  ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"


I have set 'setenv.sh' file, like this

export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"

Dose it means heap memory is still small?

回复： metastore clean OutOfMemoryError

Posted by MrWell <hu...@foxmail.com>.
Hi Mrac Wu,


Thanks, I have tried it, but it failed.&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"Marc Wu -X (mawu2 - Insigma Hengtian at Cisco)"<mawu2@cisco.com&gt;;
发送时间:&nbsp;2019年11月22日(星期五) 下午3:16
收件人:&nbsp;"user@kylin.apache.org"<user@kylin.apache.org&gt;;

主题:&nbsp;Re: metastore clean OutOfMemoryError



  
Hi MrWell, 
 
&nbsp;
 
You can try to create a file named setenv-tool.sh in the $KYLIN_HOME/conf directory, and put the following content in the file,  then execute your command. It may solve your issues.
 
#!/bin/bash
 
#
 
# Licensed to the Apache Software Foundation (ASF) under one or more
 
# contributor license agreements.&nbsp; See the NOTICE file distributed with
 
# this work for additional information regarding copyright ownership.
 
# The ASF licenses this file to You under the Apache License, Version 2.0
 
# (the "License"); you may not use this file except in compliance with
 
# the License.&nbsp; You may obtain a copy of the License at
 
#
 
#&nbsp; &nbsp; http://www.apache.org/licenses/LICENSE-2.0
 
#
 
# Unless required by applicable law or agreed to in writing, software
 
# distributed under the License is distributed on an "AS IS" BASIS,
 
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 
# See the License for the specific language governing permissions and
 
# limitations under the License.
 
# 
 
export KYLIN_EXTRA_START_OPTS="-Xmx3072M"
 
&nbsp;
 
&nbsp;
  
From: MrWell <huangpeng_scut@foxmail.com&gt;
 Reply-To: "user@kylin.apache.org" <user@kylin.apache.org&gt;
 Date: Friday, November 22, 2019 at 15:01
 To: user <user@kylin.apache.org&gt;
 Subject: metastore clean OutOfMemoryError
 
  
&nbsp;
 
  
Hi, Kylin Team. 
  
&nbsp;
 
  
When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this
 
  
&nbsp;
 
 
&nbsp;
   
java.lang.OutOfMemoryError: Java heap space
 
  
Dumping heap to java_pid4839.hprof ...
 
  
Heap dump file created [317991670 bytes in 2.120 secs]
 
  
#
 
  
# java.lang.OutOfMemoryError: Java heap space
 
  
# -XX:OnOutOfMemoryError="kill -9 %p"
 
  
#&nbsp; &nbsp;Executing /bin/sh -c "kill -9 4839"...
 
  
bin/metastore.sh: line 109:&nbsp; 4839 Killed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
 
 
  
&nbsp;
 
  
&nbsp;
 
  
I have set 'setenv.sh' file, like this
 
  
&nbsp;
 
  
export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled  -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$  -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
 
  
&nbsp;
 
  
Dose it means heap memory is still small?

Re: metastore clean OutOfMemoryError

Posted by "Marc Wu -X (mawu2 - Insigma Hengtian at Cisco)" <ma...@cisco.com>.
Hi MrWell,

You can try to create a file named setenv-tool.sh in the $KYLIN_HOME/conf directory, and put the following content in the file, then execute your command. It may solve your issues.

#!/bin/bash

#

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

#

export KYLIN_EXTRA_START_OPTS="-Xmx3072M"


From: MrWell <hu...@foxmail.com>
Reply-To: "user@kylin.apache.org" <us...@kylin.apache.org>
Date: Friday, November 22, 2019 at 15:01
To: user <us...@kylin.apache.org>
Subject: metastore clean OutOfMemoryError

Hi, Kylin Team.

When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this


java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4839.hprof ...
Heap dump file created [317991670 bytes in 2.120 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 4839"...
bin/metastore.sh: line 109:  4839 Killed                  ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"


I have set 'setenv.sh' file, like this

export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"

Dose it means heap memory is still small?

Re: metastore clean OutOfMemoryError

Posted by ShaoFeng Shi <sh...@apache.org>.
Thanks to Temple for the sharing!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




MrWell <hu...@foxmail.com> 于2019年11月22日周五 下午5:03写道：

> Hi Temple Zhou,
>
> Thanks!!!   i have solved it.  I think u r right.
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Temple Zhou"<db...@gmail.com>;
> *发送时间:* 2019年11月22日(星期五) 下午4:51
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: metastore clean OutOfMemoryError
>
> Hi MrWell,
> You should set the KYLIN_EXTRA_START_OPTS instead of KYLIN_JVM_SETTINGS
>
> exec hbase ${KYLIN_EXTRA_START_OPTS}
>> -Dkylin.hive.dependency=${hive_dependency}
>> -Dkylin.hbase.dependency=${hbase_dependency}
>> -Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties
>> "$@"
>
>
> It is *HBase Client *that became OOM because of the default small Java
> memory size.
>
> So, you can "export HBASE_OPTS="-Xmx??????"" before executing
> "bin/metastore.sh clean --delete true"
> If you are working with CDH HBase, you can increase the "-Xmx268435456" in
> /etc/hbase/conf/hbase-env.sh directly.
>
> On Fri, Nov 22, 2019 at 4:30 PM MrWell <hu...@foxmail.com> wrote:
>
>> Hi Shaofeng Shi,
>>
>> I have set KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g" in setenv.sh, but it
>> fail. Can metadata be larger than 16g?
>>
>> Thanks for reply.
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "ShaoFeng Shi"<sh...@apache.org>;
>> *发送时间:* 2019年11月22日(星期五) 下午3:18
>> *收件人:* "user"<us...@kylin.apache.org>;
>> *主题:* Re: metastore clean OutOfMemoryError
>>
>> hi Huangpeng,
>>
>> I guess your JVM heap is small or your Kylin metadata is big. You can try
>> to increase the java heap in "bin/setenv.sh"
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofengshi@apache.org
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>
>>
>>
>>
>> MrWell <hu...@foxmail.com> 于2019年11月22日周五 下午3:01写道：
>>
>>> Hi, Kylin Team.
>>>
>>> When I execute "bin/metastore.sh clean --delete true" , I get a
>>> "OutOfMemoryError" like this
>>>
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid4839.hprof ...
>>> Heap dump file created [317991670 bytes in 2.120 secs]
>>> #
>>> # java.lang.OutOfMemoryError: Java heap space
>>> # -XX:OnOutOfMemoryError="kill -9 %p"
>>> #   Executing /bin/sh -c "kill -9 4839"...
>>> bin/metastore.sh: line 109:  4839 Killed
>>> ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
>>>
>>>
>>> I have set 'setenv.sh' file, like this
>>>
>>> export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m
>>> -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4
>>> -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
>>> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc
>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>>> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
>>> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
>>>
>>> Dose it means heap memory is still small?
>>>
>>

回复： metastore clean OutOfMemoryError

Posted by MrWell <hu...@foxmail.com>.
Hi Temple Zhou,


Thanks!!!&nbsp; &nbsp;i have solved it.&nbsp; I think u r right.&nbsp;&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"Temple Zhou"<dbaztp@gmail.com&gt;;
发送时间:&nbsp;2019年11月22日(星期五) 下午4:51
收件人:&nbsp;"user"<user@kylin.apache.org&gt;;

主题:&nbsp;Re: metastore clean OutOfMemoryError



Hi MrWell,You should set the KYLIN_EXTRA_START_OPTS instead of&nbsp;KYLIN_JVM_SETTINGS


exec hbase ${KYLIN_EXTRA_START_OPTS} -Dkylin.hive.dependency=${hive_dependency} -Dkylin.hbase.dependency=${hbase_dependency} -Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties "$@"&nbsp;
It is&nbsp;HBase Client that&nbsp;became OOM because of the default small Java memory size.



So, you can "export HBASE_OPTS="-Xmx??????"" before executing "bin/metastore.sh clean --delete true"
If you are working with CDH HBase, you can increase the "-Xmx268435456" in /etc/hbase/conf/hbase-env.sh directly.


On Fri, Nov 22, 2019 at 4:30 PM MrWell <huangpeng_scut@foxmail.com&gt; wrote:

Hi Shaofeng Shi,


I have set&nbsp;KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g"&nbsp;in setenv.sh, but it fail. Can metadata be larger than 16g?


Thanks for reply.


------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"ShaoFeng Shi"<shaofengshi@apache.org&gt;;
发送时间:&nbsp;2019年11月22日(星期五) 下午3:18
收件人:&nbsp;"user"<user@kylin.apache.org&gt;;

主题:&nbsp;Re: metastore clean OutOfMemoryError



hi Huangpeng,

I guess your JVM heap is small or your Kylin metadata is big. You can try to increase the java heap in "bin/setenv.sh"

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org


Apache Kylin FAQ:&nbsp;https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org

















MrWell <huangpeng_scut@foxmail.com&gt; 于2019年11月22日周五 下午3:01写道：

Hi, Kylin Team.

When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this



java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4839.hprof ...
Heap dump file created [317991670&nbsp;bytes in 2.120 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#&nbsp; &nbsp;Executing /bin/sh -c "kill -9 4839"...
bin/metastore.sh: line 109:&nbsp; 4839 Killed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"





I have set 'setenv.sh' file, like this


export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"


Dose it means heap memory is still small?

Re: metastore clean OutOfMemoryError

Posted by Temple Zhou <db...@gmail.com>.
Hi MrWell,
You should set the KYLIN_EXTRA_START_OPTS instead of KYLIN_JVM_SETTINGS

exec hbase ${KYLIN_EXTRA_START_OPTS}
> -Dkylin.hive.dependency=${hive_dependency}
> -Dkylin.hbase.dependency=${hbase_dependency}
> -Dlog4j.configuration=file:${KYLIN_HOME}/conf/kylin-tools-log4j.properties
> "$@"


It is *HBase Client *that became OOM because of the default small Java
memory size.

So, you can "export HBASE_OPTS="-Xmx??????"" before executing
"bin/metastore.sh clean --delete true"
If you are working with CDH HBase, you can increase the "-Xmx268435456" in
/etc/hbase/conf/hbase-env.sh directly.

On Fri, Nov 22, 2019 at 4:30 PM MrWell <hu...@foxmail.com> wrote:

> Hi Shaofeng Shi,
>
> I have set KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g" in setenv.sh, but it
> fail. Can metadata be larger than 16g?
>
> Thanks for reply.
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi"<sh...@apache.org>;
> *发送时间:* 2019年11月22日(星期五) 下午3:18
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: metastore clean OutOfMemoryError
>
> hi Huangpeng,
>
> I guess your JVM heap is small or your Kylin metadata is big. You can try
> to increase the java heap in "bin/setenv.sh"
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> MrWell <hu...@foxmail.com> 于2019年11月22日周五 下午3:01写道：
>
>> Hi, Kylin Team.
>>
>> When I execute "bin/metastore.sh clean --delete true" , I get a
>> "OutOfMemoryError" like this
>>
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid4839.hprof ...
>> Heap dump file created [317991670 bytes in 2.120 secs]
>> #
>> # java.lang.OutOfMemoryError: Java heap space
>> # -XX:OnOutOfMemoryError="kill -9 %p"
>> #   Executing /bin/sh -c "kill -9 4839"...
>> bin/metastore.sh: line 109:  4839 Killed
>> ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
>>
>>
>> I have set 'setenv.sh' file, like this
>>
>> export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m
>> -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4
>> -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
>> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc
>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
>> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
>>
>> Dose it means heap memory is still small?
>>
>

回复： metastore clean OutOfMemoryError

Posted by MrWell <hu...@foxmail.com>.
Hi Shaofeng Shi,


I have set&nbsp;KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g"&nbsp;in setenv.sh, but it fail. Can metadata be larger than 16g?


Thanks for reply.


------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"ShaoFeng Shi"<shaofengshi@apache.org&gt;;
发送时间:&nbsp;2019年11月22日(星期五) 下午3:18
收件人:&nbsp;"user"<user@kylin.apache.org&gt;;

主题:&nbsp;Re: metastore clean OutOfMemoryError



hi Huangpeng,

I guess your JVM heap is small or your Kylin metadata is big. You can try to increase the java heap in "bin/setenv.sh"

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org


Apache Kylin FAQ:&nbsp;https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org

















MrWell <huangpeng_scut@foxmail.com&gt; 于2019年11月22日周五 下午3:01写道：

Hi, Kylin Team.

When I execute "bin/metastore.sh clean --delete true" , I get a "OutOfMemoryError" like this



java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4839.hprof ...
Heap dump file created [317991670&nbsp;bytes in 2.120 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#&nbsp; &nbsp;Executing /bin/sh -c "kill -9 4839"...
bin/metastore.sh: line 109:&nbsp; 4839 Killed&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"





I have set 'setenv.sh' file, like this


export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"


Dose it means heap memory is still small?

Re: metastore clean OutOfMemoryError

Posted by ShaoFeng Shi <sh...@apache.org>.
hi Huangpeng,

I guess your JVM heap is small or your Kylin metadata is big. You can try
to increase the java heap in "bin/setenv.sh"

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




MrWell <hu...@foxmail.com> 于2019年11月22日周五 下午3:01写道：

> Hi, Kylin Team.
>
> When I execute "bin/metastore.sh clean --delete true" , I get a
> "OutOfMemoryError" like this
>
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid4839.hprof ...
> Heap dump file created [317991670 bytes in 2.120 secs]
> #
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> #   Executing /bin/sh -c "kill -9 4839"...
> bin/metastore.sh: line 109:  4839 Killed
> ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.MetadataCleanupJob "${@:2}"
>
>
> I have set 'setenv.sh' file, like this
>
> export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m
> -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4
> -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
>
> Dose it means heap memory is still small?
>