You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 仇同心 <qi...@jd.com> on 2016/11/14 06:53:08 UTC

Auto merge cube error

Hi,all:
I met an error when cube was auto merge,I also set -Xmx100g -Xms100g,but this error still appeared.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
         at java.util.Arrays.copyOf(Arrays.java:2271)
         at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
         at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
         at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
         at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
         at org.apache.kylin.storage.hbase.HBaseResourceStore.putResourceImpl(HBaseResourceStore.java:239)
         at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:208)
         at org.apache.kylin.dict.DictionaryManager.save(DictionaryManager.java:413)
         at org.apache.kylin.dict.DictionaryManager.saveNewDict(DictionaryManager.java:209)
         at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:176)
         at org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:269)
         at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:145)
         at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:135)
         at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:67)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
         at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:745)


Thanks!

Re: Auto merge cube error

Posted by Li Yang <li...@apache.org>.
100g is definitely out of the scope of a normal JVM. What is your original
problem?

On Mon, Nov 14, 2016 at 2:53 PM, 仇同心 <qi...@jd.com> wrote:

> Hi,all:
>
> I met an error when cube was auto merge,I also set -Xmx100g -Xms100g,but
> this error still appeared.
>
>
>
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
>          at java.util.Arrays.copyOf(Arrays.java:2271)
>
>          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.
> java:113)
>
>          at java.io.ByteArrayOutputStream.ensureCapacity(
> ByteArrayOutputStream.java:93)
>
>          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.
> java:140)
>
>          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
>
>          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
>
>          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
>
>          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
>
>          at org.apache.kylin.storage.hbase.HBaseResourceStore.
> putResourceImpl(HBaseResourceStore.java:239)
>
>          at org.apache.kylin.common.persistence.ResourceStore.
> putResource(ResourceStore.java:208)
>
>          at org.apache.kylin.dict.DictionaryManager.save(
> DictionaryManager.java:413)
>
>          at org.apache.kylin.dict.DictionaryManager.saveNewDict(
> DictionaryManager.java:209)
>
>          at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(
> DictionaryManager.java:176)
>
>          at org.apache.kylin.dict.DictionaryManager.mergeDictionary(
> DictionaryManager.java:269)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> mergeDictionaries(MergeDictionaryStep.java:145)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> makeDictForNewSegment(MergeDictionaryStep.java:135)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> doWork(MergeDictionaryStep.java:67)
>
>          at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>          at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(DefaultChainedExecutable.java:57)
>
>          at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>          at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
>
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Thanks!
>

Re: Auto merge cube error

Posted by Li Yang <li...@apache.org>.
100g is definitely out of the scope of a normal JVM. What is your original
problem?

On Mon, Nov 14, 2016 at 2:53 PM, 仇同心 <qi...@jd.com> wrote:

> Hi,all:
>
> I met an error when cube was auto merge,I also set -Xmx100g -Xms100g,but
> this error still appeared.
>
>
>
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
>          at java.util.Arrays.copyOf(Arrays.java:2271)
>
>          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.
> java:113)
>
>          at java.io.ByteArrayOutputStream.ensureCapacity(
> ByteArrayOutputStream.java:93)
>
>          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.
> java:140)
>
>          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
>
>          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
>
>          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
>
>          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
>
>          at org.apache.kylin.storage.hbase.HBaseResourceStore.
> putResourceImpl(HBaseResourceStore.java:239)
>
>          at org.apache.kylin.common.persistence.ResourceStore.
> putResource(ResourceStore.java:208)
>
>          at org.apache.kylin.dict.DictionaryManager.save(
> DictionaryManager.java:413)
>
>          at org.apache.kylin.dict.DictionaryManager.saveNewDict(
> DictionaryManager.java:209)
>
>          at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(
> DictionaryManager.java:176)
>
>          at org.apache.kylin.dict.DictionaryManager.mergeDictionary(
> DictionaryManager.java:269)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> mergeDictionaries(MergeDictionaryStep.java:145)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> makeDictForNewSegment(MergeDictionaryStep.java:135)
>
>          at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.
> doWork(MergeDictionaryStep.java:67)
>
>          at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>          at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(DefaultChainedExecutable.java:57)
>
>          at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
>
>          at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
>
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> Thanks!
>