You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Dafeng Wang <da...@microsoft.com> on 2016/02/25 03:30:45 UTC
OOM error when run all paragraphs
Hi,
Today when I try to run the zeppelin with a complicated notebook, if I run them one by one, it works well, however, when I click on running all paragraphs, it will throw OOM errors in a short time, I set zeppelin’s interpreter’s memory as 6G. any idea?
Regards,
Dafeng
Re: OOM error when run all paragraphs
Posted by moon soo Lee <mo...@apache.org>.
Hi Dafeng,
Thanks for sharing your verification!
Thanks,
moon
On Wed, Mar 9, 2016 at 6:33 PM Dafeng Wang <da...@microsoft.com> wrote:
> Hi Moon,
>
>
>
> I test it locally, it passed
>
> 1. No oom happened after running notebook
>
> 2. Memory consumption is 1/6 of original version
>
>
>
> Regards,
>
> Dafeng
>
>
>
> *From:* Dafeng Wang [mailto:dafewan@microsoft.com]
> *Sent:* Tuesday, March 8, 2016 4:14 PM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* RE: OOM error when run all paragraphs
>
>
>
> I’ll do a verify, then post the result.
>
>
>
> *From:* moon soo Lee [mailto:moon@apache.org <mo...@apache.org>]
> *Sent:* Saturday, March 5, 2016 9:28 AM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* Re: OOM error when run all paragraphs
>
>
>
> Hi,
>
>
>
> I just created a patch.
>
> Please see https://github.com/apache/incubator-zeppelin/pull/762
>
>
>
> Thanks,
>
> moon
>
>
>
> On Mon, Feb 29, 2016 at 10:01 PM Skanda <sk...@gmail.com>
> wrote:
>
> Hi
>
>
>
> Im also hitting this issue of OOM for couple of notebooks with about 6 to
> 7 paragraphs running Hive queries and caching 1000 records of output.
>
>
>
> Regards,
>
> Skanda
>
>
>
> On Mon, Feb 29, 2016 at 10:42 PM, moon soo Lee <mo...@apache.org> wrote:
>
> Thanks for creating an issue.
>
> Let me look into it more.
>
>
>
> Best,
>
> moon
>
>
>
> On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com>
> wrote:
>
> Thanks Moon for your effort to repro this bug, I had create a Jira
> https://issues.apache.org/jira/browse/ZEPPELIN-706
> <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fZEPPELIN-706&data=01%7c01%7cdafewan%40064d.mgd.microsoft.com%7c15e6d6d68c6b48ea143a08d3449577fb%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2fML9U5kSzqgJjmOP4Beci87nA8tIeoCORMcja%2bTUhBw%3d>
> for it, please let me know if you need anything else.
>
>
>
> Regards,
>
> Dafeng
>
>
>
> *From:* moon soo Lee [mailto:moon@apache.org]
> *Sent:* Saturday, February 27, 2016 2:25 AM
>
>
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* Re: OOM error when run all paragraphs
>
>
>
> Thanks for sharing your use case and the memory usage table.
> I could able to reproduce the problem. Do you mind create an issue for it
> on our jira?
> I might have sometime next week to dig into this problem.
>
> Thanks,
> moon
>
>
>
> On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com> wrote:
>
> Hi Moon,
>
>
>
> Thanks for your reply, as for my case: my zeppelin server only have 1
> notebook with almost all queries are sparkSql query(30 paragraphs), the
> result limitation I set is 10000, all query reach the limitation of 10000.
>
> The strange thing is: run them one by one won’t cause the OOM error, if I
> click on run notebooks, it will OOM quickly.
>
>
>
> I check on the memory usage, the fastest increasing part is [B and [C,
> however, it just tells me we do have memory increase, doesn’t hint any
> other things, I put the table below for reference, the last one is the
> latest one, you can see only *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8> *got increased when I run
> it again and again, no obvious clue of memory leak.
>
>
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1090
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1091
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 115
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 121
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 34
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 36
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>
>
RE: OOM error when run all paragraphs
Posted by Dafeng Wang <da...@microsoft.com>.
Hi Moon,
I test it locally, it passed
1. No oom happened after running notebook
2. Memory consumption is 1/6 of original version
Regards,
Dafeng
From: Dafeng Wang [mailto:dafewan@microsoft.com]
Sent: Tuesday, March 8, 2016 4:14 PM
To: users@zeppelin.incubator.apache.org
Subject: RE: OOM error when run all paragraphs
I’ll do a verify, then post the result.
From: moon soo Lee [mailto:moon@apache.org]
Sent: Saturday, March 5, 2016 9:28 AM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Hi,
I just created a patch.
Please see https://github.com/apache/incubator-zeppelin/pull/762
Thanks,
moon
On Mon, Feb 29, 2016 at 10:01 PM Skanda <sk...@gmail.com>> wrote:
Hi
Im also hitting this issue of OOM for couple of notebooks with about 6 to 7 paragraphs running Hive queries and caching 1000 records of output.
Regards,
Skanda
On Mon, Feb 29, 2016 at 10:42 PM, moon soo Lee <mo...@apache.org>> wrote:
Thanks for creating an issue.
Let me look into it more.
Best,
moon
On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com>> wrote:
Thanks Moon for your effort to repro this bug, I had create a Jira https://issues.apache.org/jira/browse/ZEPPELIN-706<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fZEPPELIN-706&data=01%7c01%7cdafewan%40064d.mgd.microsoft.com%7c15e6d6d68c6b48ea143a08d3449577fb%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2fML9U5kSzqgJjmOP4Beci87nA8tIeoCORMcja%2bTUhBw%3d> for it, please let me know if you need anything else.
Regards,
Dafeng
From: moon soo Lee [mailto:moon@apache.org<ma...@apache.org>]
Sent: Saturday, February 27, 2016 2:25 AM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Thanks for sharing your use case and the memory usage table.
I could able to reproduce the problem. Do you mind create an issue for it on our jira?
I might have sometime next week to dig into this problem.
Thanks,
moon
On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com>> wrote:
Hi Moon,
Thanks for your reply, as for my case: my zeppelin server only have 1 notebook with almost all queries are sparkSql query(30 paragraphs), the result limitation I set is 10000, all query reach the limitation of 10000.
The strange thing is: run them one by one won’t cause the OOM error, if I click on run notebooks, it will OOM quickly.
I check on the memory usage, the fastest increasing part is [B and [C, however, it just tells me we do have memory increase, doesn’t hint any other things, I put the table below for reference, the last one is the latest one, you can see only scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> got increased when I run it again and again, no obvious clue of memory leak.
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1090
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1091
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
[[B<http://localhost:7000/refsByType/0x616897178>
150
[[B<http://localhost:7000/refsByType/0x616897178>
150
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
115
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
121
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
34
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
36
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
From: moon soo Lee [mailto:moon@apache.org<ma...@apache.org>]
Sent: Thursday, February 25, 2016 10:56 PM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Hi Dafeng,
Zeppelin keeps notebooks in memory.
That may cause OOM when notebooks has a lot of con
RE: OOM error when run all paragraphs
Posted by Dafeng Wang <da...@microsoft.com>.
I’ll do a verify, then post the result.
From: moon soo Lee [mailto:moon@apache.org]
Sent: Saturday, March 5, 2016 9:28 AM
To: users@zeppelin.incubator.apache.org
Subject: Re: OOM error when run all paragraphs
Hi,
I just created a patch.
Please see https://github.com/apache/incubator-zeppelin/pull/762
Thanks,
moon
On Mon, Feb 29, 2016 at 10:01 PM Skanda <sk...@gmail.com>> wrote:
Hi
Im also hitting this issue of OOM for couple of notebooks with about 6 to 7 paragraphs running Hive queries and caching 1000 records of output.
Regards,
Skanda
On Mon, Feb 29, 2016 at 10:42 PM, moon soo Lee <mo...@apache.org>> wrote:
Thanks for creating an issue.
Let me look into it more.
Best,
moon
On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com>> wrote:
Thanks Moon for your effort to repro this bug, I had create a Jira https://issues.apache.org/jira/browse/ZEPPELIN-706<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fZEPPELIN-706&data=01%7c01%7cdafewan%40064d.mgd.microsoft.com%7c15e6d6d68c6b48ea143a08d3449577fb%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2fML9U5kSzqgJjmOP4Beci87nA8tIeoCORMcja%2bTUhBw%3d> for it, please let me know if you need anything else.
Regards,
Dafeng
From: moon soo Lee [mailto:moon@apache.org<ma...@apache.org>]
Sent: Saturday, February 27, 2016 2:25 AM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Thanks for sharing your use case and the memory usage table.
I could able to reproduce the problem. Do you mind create an issue for it on our jira?
I might have sometime next week to dig into this problem.
Thanks,
moon
On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com>> wrote:
Hi Moon,
Thanks for your reply, as for my case: my zeppelin server only have 1 notebook with almost all queries are sparkSql query(30 paragraphs), the result limitation I set is 10000, all query reach the limitation of 10000.
The strange thing is: run them one by one won’t cause the OOM error, if I click on run notebooks, it will OOM quickly.
I check on the memory usage, the fastest increasing part is [B and [C, however, it just tells me we do have memory increase, doesn’t hint any other things, I put the table below for reference, the last one is the latest one, you can see only scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> got increased when I run it again and again, no obvious clue of memory leak.
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1090
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1091
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
[[B<http://localhost:7000/refsByType/0x616897178>
150
[[B<http://localhost:7000/refsByType/0x616897178>
150
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
115
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
121
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
34
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
36
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
From: moon soo Lee [mailto:moon@apache.org<ma...@apache.org>]
Sent: Thursday, February 25, 2016 10:56 PM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Hi Dafeng,
Zeppelin keeps notebooks in memory.
That may cause OOM when notebooks has a lot of con
Re: OOM error when run all paragraphs
Posted by moon soo Lee <mo...@apache.org>.
Hi,
I just created a patch.
Please see https://github.com/apache/incubator-zeppelin/pull/762
Thanks,
moon
On Mon, Feb 29, 2016 at 10:01 PM Skanda <sk...@gmail.com> wrote:
> Hi
>
> Im also hitting this issue of OOM for couple of notebooks with about 6 to
> 7 paragraphs running Hive queries and caching 1000 records of output.
>
> Regards,
> Skanda
>
> On Mon, Feb 29, 2016 at 10:42 PM, moon soo Lee <mo...@apache.org> wrote:
>
>> Thanks for creating an issue.
>> Let me look into it more.
>>
>> Best,
>> moon
>>
>> On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com>
>> wrote:
>>
>>> Thanks Moon for your effort to repro this bug, I had create a Jira
>>> https://issues.apache.org/jira/browse/ZEPPELIN-706 for it, please let
>>> me know if you need anything else.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Dafeng
>>>
>>>
>>>
>>> *From:* moon soo Lee [mailto:moon@apache.org]
>>> *Sent:* Saturday, February 27, 2016 2:25 AM
>>>
>>>
>>> *To:* users@zeppelin.incubator.apache.org
>>> *Subject:* Re: OOM error when run all paragraphs
>>>
>>>
>>>
>>> Thanks for sharing your use case and the memory usage table.
>>> I could able to reproduce the problem. Do you mind create an issue for
>>> it on our jira?
>>> I might have sometime next week to dig into this problem.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>>
>>> On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com>
>>> wrote:
>>>
>>> Hi Moon,
>>>
>>>
>>>
>>> Thanks for your reply, as for my case: my zeppelin server only have 1
>>> notebook with almost all queries are sparkSql query(30 paragraphs), the
>>> result limitation I set is 10000, all query reach the limitation of 10000.
>>>
>>> The strange thing is: run them one by one won’t cause the OOM error, if
>>> I click on run notebooks, it will OOM quickly.
>>>
>>>
>>>
>>> I check on the memory usage, the fastest increasing part is [B and [C,
>>> however, it just tells me we do have memory increase, doesn’t hint any
>>> other things, I put the table below for reference, the last one is the
>>> latest one, you can see only *scala.reflect.io.VirtualFile
>>> <http://localhost:7000/refsByType/0x616022bc8> *got increased when I
>>> run it again and again, no obvious clue of memory leak.
>>>
>>>
>>>
>>> *scala.reflect.io.FileZipArchive$FileEntry$1
>>> <http://localhost:7000/refsByType/0x616022ae8>*
>>>
>>> 224886
>>>
>>> *scala.reflect.io.FileZipArchive$FileEntry$1
>>> <http://localhost:7000/refsByType/0x616022ae8>*
>>>
>>> 224886
>>>
>>> *scala.reflect.io.ZipArchive$DirEntry
>>> <http://localhost:7000/refsByType/0x616022ca8>*
>>>
>>> 10123
>>>
>>> *scala.reflect.io.ZipArchive$DirEntry
>>> <http://localhost:7000/refsByType/0x616022ca8>*
>>>
>>> 10123
>>>
>>> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>>>
>>> 1090
>>>
>>> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>>>
>>> 1091
>>>
>>> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>>>
>>> 960
>>>
>>> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>>>
>>> 960
>>>
>>> *org.apache.derby.iapi.services.io.FormatIdInputStream
>>> <http://localhost:7000/refsByType/0x61b38f2a0>*
>>>
>>> 884
>>>
>>> *org.apache.derby.iapi.services.io.FormatIdInputStream
>>> <http://localhost:7000/refsByType/0x61b38f2a0>*
>>>
>>> 884
>>>
>>> *sun.security.util.DerInputBuffer
>>> <http://localhost:7000/refsByType/0x61684fc40>*
>>>
>>> 745
>>>
>>> *sun.security.util.DerInputBuffer
>>> <http://localhost:7000/refsByType/0x61684fc40>*
>>>
>>> 745
>>>
>>> *org.apache.derby.iapi.services.io.FormatIdOutputStream
>>> <http://localhost:7000/refsByType/0x61b3f5f20>*
>>>
>>> 479
>>>
>>> *org.apache.derby.iapi.services.io.FormatIdOutputStream
>>> <http://localhost:7000/refsByType/0x61b3f5f20>*
>>>
>>> 479
>>>
>>> *sun.security.util.ObjectIdentifier
>>> <http://localhost:7000/refsByType/0x615d02320>*
>>>
>>> 466
>>>
>>> *sun.security.util.ObjectIdentifier
>>> <http://localhost:7000/refsByType/0x615d02320>*
>>>
>>> 466
>>>
>>> *org.apache.derby.iapi.services.io.ArrayInputStream
>>> <http://localhost:7000/refsByType/0x61b215910>*
>>>
>>> 442
>>>
>>> *org.apache.derby.iapi.services.io.ArrayInputStream
>>> <http://localhost:7000/refsByType/0x61b215910>*
>>>
>>> 442
>>>
>>> *org.apache.derby.iapi.services.io.ArrayOutputStream
>>> <http://localhost:7000/refsByType/0x61b3f6000>*
>>>
>>> 442
>>>
>>> *org.apache.derby.iapi.services.io.ArrayOutputStream
>>> <http://localhost:7000/refsByType/0x61b3f6000>*
>>>
>>> 442
>>>
>>> *org.apache.derby.iapi.services.io.FormatableBitSet
>>> <http://localhost:7000/refsByType/0x617ae17e8>*
>>>
>>> 364
>>>
>>> *org.apache.derby.iapi.services.io.FormatableBitSet
>>> <http://localhost:7000/refsByType/0x617ae17e8>*
>>>
>>> 364
>>>
>>> *scala.reflect.internal.pickling.UnPickler$Scan
>>> <http://localhost:7000/refsByType/0x6166d6e20>*
>>>
>>> 282
>>>
>>> *scala.reflect.internal.pickling.UnPickler$Scan
>>> <http://localhost:7000/refsByType/0x6166d6e20>*
>>>
>>> 282
>>>
>>> *org.apache.derby.impl.store.raw.data.StoredPage
>>> <http://localhost:7000/refsByType/0x61ba57d88>*
>>>
>>> 279
>>>
>>> *org.apache.derby.impl.store.raw.data.StoredPage
>>> <http://localhost:7000/refsByType/0x61ba57d88>*
>>>
>>> 279
>>>
>>> *java.util.jar.JarFile$JarFileEntry
>>> <http://localhost:7000/refsByType/0x615e24618>*
>>>
>>> 276
>>>
>>> *java.util.jar.JarFile$JarFileEntry
>>> <http://localhost:7000/refsByType/0x615e24618>*
>>>
>>> 276
>>>
>>> *sun.security.x509.X509CertImpl
>>> <http://localhost:7000/refsByType/0x616083cf0>*
>>>
>>> 166
>>>
>>> *sun.security.x509.X509CertImpl
>>> <http://localhost:7000/refsByType/0x616083cf0>*
>>>
>>> 166
>>>
>>> *org.apache.derby.impl.store.raw.data.AllocPage
>>> <http://localhost:7000/refsByType/0x61b15ef28>*
>>>
>>> 162
>>>
>>> *org.apache.derby.impl.store.raw.data.AllocPage
>>> <http://localhost:7000/refsByType/0x61b15ef28>*
>>>
>>> 162
>>>
>>> *org.spark-project.jetty.io.BufferCache$CachedBuffer
>>> <http://localhost:7000/refsByType/0x616325310>*
>>>
>>> 155
>>>
>>> *org.spark-project.jetty.io.BufferCache$CachedBuffer
>>> <http://localhost:7000/refsByType/0x616325310>*
>>>
>>> 155
>>>
>>> *[[B <http://localhost:7000/refsByType/0x616897178>*
>>>
>>> 150
>>>
>>> *[[B <http://localhost:7000/refsByType/0x616897178>*
>>>
>>> 150
>>>
>>> *org.spark-project.jetty.io.ByteArrayBuffer
>>> <http://localhost:7000/refsByType/0x616309b60>*
>>>
>>> 138
>>>
>>> *org.spark-project.jetty.io.ByteArrayBuffer
>>> <http://localhost:7000/refsByType/0x616309b60>*
>>>
>>> 138
>>>
>>> *org.apache.thrift.protocol.TBinaryProtocol
>>> <http://localhost:7000/refsByType/0x615f91410>*
>>>
>>> 128
>>>
>>> *org.apache.thrift.protocol.TBinaryProtocol
>>> <http://localhost:7000/refsByType/0x615f91410>*
>>>
>>> 128
>>>
>>> *scala.reflect.io.VirtualFile
>>> <http://localhost:7000/refsByType/0x616022bc8>*
>>>
>>> 115
>>>
>>> *scala.reflect.io.VirtualFile
>>> <http://localhost:7000/refsByType/0x616022bc8>*
>>>
>>> 121
>>>
>>> *org.apache.derby.impl.store.raw.data.RAFContainer4
>>> <http://localhost:7000/refsByType/0x61b974dc0>*
>>>
>>> 100
>>>
>>> *org.apache.derby.impl.store.raw.data.RAFContainer4
>>> <http://localhost:7000/refsByType/0x61b974dc0>*
>>>
>>> 100
>>>
>>> *sun.security.x509.KeyIdentifier
>>> <http://localhost:7000/refsByType/0x616856000>*
>>>
>>> 95
>>>
>>> *sun.security.x509.KeyIdentifier
>>> <http://localhost:7000/refsByType/0x616856000>*
>>>
>>> 95
>>>
>>> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>>>
>>> 94
>>>
>>> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>>>
>>> 94
>>>
>>> *sun.security.util.Cache$EqualByteArray
>>> <http://localhost:7000/refsByType/0x61688c258>*
>>>
>>> 84
>>>
>>> *sun.security.util.Cache$EqualByteArray
>>> <http://localhost:7000/refsByType/0x61688c258>*
>>>
>>> 84
>>>
>>> *sun.security.x509.X509CertInfo
>>> <http://localhost:7000/refsByType/0x6163a8398>*
>>>
>>> 83
>>>
>>> *sun.security.util.BitArray
>>> <http://localhost:7000/refsByType/0x61683bdb0>*
>>>
>>> 83
>>>
>>> *sun.security.util.BitArray
>>> <http://localhost:7000/refsByType/0x61683bdb0>*
>>>
>>> 83
>>>
>>> *sun.security.x509.X509CertInfo
>>> <http://localhost:7000/refsByType/0x6163a8398>*
>>>
>>> 83
>>>
>>> *sun.security.rsa.RSAPublicKeyImpl
>>> <http://localhost:7000/refsByType/0x6160e75d0>*
>>>
>>> 80
>>>
>>> *sun.security.rsa.RSAPublicKeyImpl
>>> <http://localhost:7000/refsByType/0x6160e75d0>*
>>>
>>> 80
>>>
>>> *sun.security.x509.BasicConstraintsExtension
>>> <http://localhost:7000/refsByType/0x616568910>*
>>>
>>> 72
>>>
>>> *sun.security.x509.BasicConstraintsExtension
>>> <http://localhost:7000/refsByType/0x616568910>*
>>>
>>> 72
>>>
>>> *sun.security.x509.SubjectKeyIdentifierExtension
>>> <http://localhost:7000/refsByType/0x616568890>*
>>>
>>> 65
>>>
>>> *sun.security.x509.SubjectKeyIdentifierExtension
>>> <http://localhost:7000/refsByType/0x616568890>*
>>>
>>> 65
>>>
>>> *sun.security.x509.KeyUsageExtension
>>> <http://localhost:7000/refsByType/0x615e96ec0>*
>>>
>>> 62
>>>
>>> *sun.security.x509.KeyUsageExtension
>>> <http://localhost:7000/refsByType/0x615e96ec0>*
>>>
>>> 62
>>>
>>> *sun.security.ec.NamedCurve
>>> <http://localhost:7000/refsByType/0x615e28ef0>*
>>>
>>> 46
>>>
>>> *sun.security.ec.NamedCurve
>>> <http://localhost:7000/refsByType/0x615e28ef0>*
>>>
>>> 46
>>>
>>> *io.netty.buffer.PoolChunk
>>> <http://localhost:7000/refsByType/0x61c0559a0>*
>>>
>>> 40
>>>
>>> *io.netty.buffer.PoolChunk
>>> <http://localhost:7000/refsByType/0x61c0559a0>*
>>>
>>> 40
>>>
>>> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
>>> <http://localhost:7000/refsByType/0x61637b1a0>*
>>>
>>> 34
>>>
>>> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
>>> <http://localhost:7000/refsByType/0x61637b1a0>*
>>>
>>> 36
>>>
>>> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>>>
>>> 32
>>>
>>> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>>>
>>> 32
>>>
>>> *sun.security.x509.AuthorityKeyIdentifierExtension
>>> <http://localhost:7000/refsByType/0x615f917c8>*
>>>
>>> 30
>>>
>>> *sun.security.x509.AuthorityKeyIdentifierExtension
>>> <http://localhost:7000/refsByType/0x615f917c8>*
>>>
>>> 30
>>>
>>> *java.security.cert.PolicyQualifierInfo
>>> <http://localhost:7000/refsByType/0x61672f090>*
>>>
>>> 28
>>>
>>> *java.security.cert.PolicyQualifierInfo
>>> <http://localhost:7000/refsByType/0x61672f090>*
>>>
>>> 28
>>>
>>> *java.io.ByteArrayOutputStream
>>> <http://localhost:7000/refsByType/0x6162c3f98>*
>>>
>>> 24
>>>
>>> *java.io.ByteArrayOutputStream
>>> <http://localhost:7000/refsByType/0x6162c3f98>*
>>>
>>> 24
>>>
>>> *java.util.jar.JarVerifier
>>> <http://localhost:7000/refsByType/0x615fc7838>*
>>>
>>> 22
>>>
>>> *java.util.jar.JarVerifier
>>> <http://localhost:7000/refsByType/0x615fc7838>*
>>>
>>> 22
>>>
>>> *sun.security.x509.CRLDistributionPointsExtension
>>> <http://localhost:7000/refsByType/0x616568810>*
>>>
>>> 17
>>>
>>> *sun.security.x509.CRLDistributionPointsExtension
>>> <http://localhost:7000/refsByType/0x616568810>*
>>>
>>> 17
>>>
>>> *java.io.BufferedOutputStream
>>> <http://localhost:7000/refsByType/0x61623db10>*
>>>
>>> 13
>>>
>>> *java.io.BufferedOutputStream
>>> <http://localhost:7000/refsByType/0x61623db10>*
>>>
>>> 13
>>>
>>> *java.net.SocketOutputStream
>>> <http://localhost:7000/refsByType/0x61665ded8>*
>>>
>>> 12
>>>
>>> *java.net.SocketOutputStream
>>> <http://localhost:7000/refsByType/0x61665ded8>*
>>>
>>> 12
>>>
>>> *sun.security.provider.SHA
>>> <http://localhost:7000/refsByType/0x615e756c0>*
>>>
>>> 12
>>>
>>> *sun.security.x509.CertificatePoliciesExtension
>>> <http://localhost:7000/refsByType/0x616568710>*
>>>
>>> 12
>>>
>>> *sun.security.x509.CertificatePoliciesExtension
>>> <http://localhost:7000/refsByType/0x616568710>*
>>>
>>> 12
>>>
>>> *sun.security.provider.SHA
>>> <http://localhost:7000/refsByType/0x615e756c0>*
>>>
>>> 12
>>>
>>> *java.io.BufferedInputStream
>>> <http://localhost:7000/refsByType/0x61669a630>*
>>>
>>> 10
>>>
>>> *java.io.BufferedInputStream
>>> <http://localhost:7000/refsByType/0x61669a630>*
>>>
>>> 10
>>>
>>> *sun.security.provider.SecureRandom
>>> <http://localhost:7000/refsByType/0x615f128c8>*
>>>
>>> 10
>>>
>>> *sun.security.provider.SecureRandom
>>> <http://localhost:7000/refsByType/0x615f128c8>*
>>>
>>> 10
>>>
>>>
>>>
>>>
>>>
>>> *From:* moon soo Lee [mailto:moon@apache.org]
>>> *Sent:* Thursday, February 25, 2016 10:56 PM
>>> *To:* users@zeppelin.incubator.apache.org
>>> *Subject:* Re: OOM error when run all paragraphs
>>>
>>>
>>>
>>> Hi Dafeng,
>>>
>>>
>>>
>>> Zeppelin keeps notebooks in memory.
>>>
>>> That may cause OOM when notebooks has a lot of con
>>>
>>>
>
Re: OOM error when run all paragraphs
Posted by Skanda <sk...@gmail.com>.
Hi
Im also hitting this issue of OOM for couple of notebooks with about 6 to 7
paragraphs running Hive queries and caching 1000 records of output.
Regards,
Skanda
On Mon, Feb 29, 2016 at 10:42 PM, moon soo Lee <mo...@apache.org> wrote:
> Thanks for creating an issue.
> Let me look into it more.
>
> Best,
> moon
>
> On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com>
> wrote:
>
>> Thanks Moon for your effort to repro this bug, I had create a Jira
>> https://issues.apache.org/jira/browse/ZEPPELIN-706 for it, please let me
>> know if you need anything else.
>>
>>
>>
>> Regards,
>>
>> Dafeng
>>
>>
>>
>> *From:* moon soo Lee [mailto:moon@apache.org]
>> *Sent:* Saturday, February 27, 2016 2:25 AM
>>
>>
>> *To:* users@zeppelin.incubator.apache.org
>> *Subject:* Re: OOM error when run all paragraphs
>>
>>
>>
>> Thanks for sharing your use case and the memory usage table.
>> I could able to reproduce the problem. Do you mind create an issue for it
>> on our jira?
>> I might have sometime next week to dig into this problem.
>>
>> Thanks,
>> moon
>>
>>
>>
>> On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com>
>> wrote:
>>
>> Hi Moon,
>>
>>
>>
>> Thanks for your reply, as for my case: my zeppelin server only have 1
>> notebook with almost all queries are sparkSql query(30 paragraphs), the
>> result limitation I set is 10000, all query reach the limitation of 10000.
>>
>> The strange thing is: run them one by one won’t cause the OOM error, if I
>> click on run notebooks, it will OOM quickly.
>>
>>
>>
>> I check on the memory usage, the fastest increasing part is [B and [C,
>> however, it just tells me we do have memory increase, doesn’t hint any
>> other things, I put the table below for reference, the last one is the
>> latest one, you can see only *scala.reflect.io.VirtualFile
>> <http://localhost:7000/refsByType/0x616022bc8> *got increased when I run
>> it again and again, no obvious clue of memory leak.
>>
>>
>>
>> *scala.reflect.io.FileZipArchive$FileEntry$1
>> <http://localhost:7000/refsByType/0x616022ae8>*
>>
>> 224886
>>
>> *scala.reflect.io.FileZipArchive$FileEntry$1
>> <http://localhost:7000/refsByType/0x616022ae8>*
>>
>> 224886
>>
>> *scala.reflect.io.ZipArchive$DirEntry
>> <http://localhost:7000/refsByType/0x616022ca8>*
>>
>> 10123
>>
>> *scala.reflect.io.ZipArchive$DirEntry
>> <http://localhost:7000/refsByType/0x616022ca8>*
>>
>> 10123
>>
>> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>>
>> 1090
>>
>> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>>
>> 1091
>>
>> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>>
>> 960
>>
>> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>>
>> 960
>>
>> *org.apache.derby.iapi.services.io.FormatIdInputStream
>> <http://localhost:7000/refsByType/0x61b38f2a0>*
>>
>> 884
>>
>> *org.apache.derby.iapi.services.io.FormatIdInputStream
>> <http://localhost:7000/refsByType/0x61b38f2a0>*
>>
>> 884
>>
>> *sun.security.util.DerInputBuffer
>> <http://localhost:7000/refsByType/0x61684fc40>*
>>
>> 745
>>
>> *sun.security.util.DerInputBuffer
>> <http://localhost:7000/refsByType/0x61684fc40>*
>>
>> 745
>>
>> *org.apache.derby.iapi.services.io.FormatIdOutputStream
>> <http://localhost:7000/refsByType/0x61b3f5f20>*
>>
>> 479
>>
>> *org.apache.derby.iapi.services.io.FormatIdOutputStream
>> <http://localhost:7000/refsByType/0x61b3f5f20>*
>>
>> 479
>>
>> *sun.security.util.ObjectIdentifier
>> <http://localhost:7000/refsByType/0x615d02320>*
>>
>> 466
>>
>> *sun.security.util.ObjectIdentifier
>> <http://localhost:7000/refsByType/0x615d02320>*
>>
>> 466
>>
>> *org.apache.derby.iapi.services.io.ArrayInputStream
>> <http://localhost:7000/refsByType/0x61b215910>*
>>
>> 442
>>
>> *org.apache.derby.iapi.services.io.ArrayInputStream
>> <http://localhost:7000/refsByType/0x61b215910>*
>>
>> 442
>>
>> *org.apache.derby.iapi.services.io.ArrayOutputStream
>> <http://localhost:7000/refsByType/0x61b3f6000>*
>>
>> 442
>>
>> *org.apache.derby.iapi.services.io.ArrayOutputStream
>> <http://localhost:7000/refsByType/0x61b3f6000>*
>>
>> 442
>>
>> *org.apache.derby.iapi.services.io.FormatableBitSet
>> <http://localhost:7000/refsByType/0x617ae17e8>*
>>
>> 364
>>
>> *org.apache.derby.iapi.services.io.FormatableBitSet
>> <http://localhost:7000/refsByType/0x617ae17e8>*
>>
>> 364
>>
>> *scala.reflect.internal.pickling.UnPickler$Scan
>> <http://localhost:7000/refsByType/0x6166d6e20>*
>>
>> 282
>>
>> *scala.reflect.internal.pickling.UnPickler$Scan
>> <http://localhost:7000/refsByType/0x6166d6e20>*
>>
>> 282
>>
>> *org.apache.derby.impl.store.raw.data.StoredPage
>> <http://localhost:7000/refsByType/0x61ba57d88>*
>>
>> 279
>>
>> *org.apache.derby.impl.store.raw.data.StoredPage
>> <http://localhost:7000/refsByType/0x61ba57d88>*
>>
>> 279
>>
>> *java.util.jar.JarFile$JarFileEntry
>> <http://localhost:7000/refsByType/0x615e24618>*
>>
>> 276
>>
>> *java.util.jar.JarFile$JarFileEntry
>> <http://localhost:7000/refsByType/0x615e24618>*
>>
>> 276
>>
>> *sun.security.x509.X509CertImpl
>> <http://localhost:7000/refsByType/0x616083cf0>*
>>
>> 166
>>
>> *sun.security.x509.X509CertImpl
>> <http://localhost:7000/refsByType/0x616083cf0>*
>>
>> 166
>>
>> *org.apache.derby.impl.store.raw.data.AllocPage
>> <http://localhost:7000/refsByType/0x61b15ef28>*
>>
>> 162
>>
>> *org.apache.derby.impl.store.raw.data.AllocPage
>> <http://localhost:7000/refsByType/0x61b15ef28>*
>>
>> 162
>>
>> *org.spark-project.jetty.io.BufferCache$CachedBuffer
>> <http://localhost:7000/refsByType/0x616325310>*
>>
>> 155
>>
>> *org.spark-project.jetty.io.BufferCache$CachedBuffer
>> <http://localhost:7000/refsByType/0x616325310>*
>>
>> 155
>>
>> *[[B <http://localhost:7000/refsByType/0x616897178>*
>>
>> 150
>>
>> *[[B <http://localhost:7000/refsByType/0x616897178>*
>>
>> 150
>>
>> *org.spark-project.jetty.io.ByteArrayBuffer
>> <http://localhost:7000/refsByType/0x616309b60>*
>>
>> 138
>>
>> *org.spark-project.jetty.io.ByteArrayBuffer
>> <http://localhost:7000/refsByType/0x616309b60>*
>>
>> 138
>>
>> *org.apache.thrift.protocol.TBinaryProtocol
>> <http://localhost:7000/refsByType/0x615f91410>*
>>
>> 128
>>
>> *org.apache.thrift.protocol.TBinaryProtocol
>> <http://localhost:7000/refsByType/0x615f91410>*
>>
>> 128
>>
>> *scala.reflect.io.VirtualFile
>> <http://localhost:7000/refsByType/0x616022bc8>*
>>
>> 115
>>
>> *scala.reflect.io.VirtualFile
>> <http://localhost:7000/refsByType/0x616022bc8>*
>>
>> 121
>>
>> *org.apache.derby.impl.store.raw.data.RAFContainer4
>> <http://localhost:7000/refsByType/0x61b974dc0>*
>>
>> 100
>>
>> *org.apache.derby.impl.store.raw.data.RAFContainer4
>> <http://localhost:7000/refsByType/0x61b974dc0>*
>>
>> 100
>>
>> *sun.security.x509.KeyIdentifier
>> <http://localhost:7000/refsByType/0x616856000>*
>>
>> 95
>>
>> *sun.security.x509.KeyIdentifier
>> <http://localhost:7000/refsByType/0x616856000>*
>>
>> 95
>>
>> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>>
>> 94
>>
>> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>>
>> 94
>>
>> *sun.security.util.Cache$EqualByteArray
>> <http://localhost:7000/refsByType/0x61688c258>*
>>
>> 84
>>
>> *sun.security.util.Cache$EqualByteArray
>> <http://localhost:7000/refsByType/0x61688c258>*
>>
>> 84
>>
>> *sun.security.x509.X509CertInfo
>> <http://localhost:7000/refsByType/0x6163a8398>*
>>
>> 83
>>
>> *sun.security.util.BitArray
>> <http://localhost:7000/refsByType/0x61683bdb0>*
>>
>> 83
>>
>> *sun.security.util.BitArray
>> <http://localhost:7000/refsByType/0x61683bdb0>*
>>
>> 83
>>
>> *sun.security.x509.X509CertInfo
>> <http://localhost:7000/refsByType/0x6163a8398>*
>>
>> 83
>>
>> *sun.security.rsa.RSAPublicKeyImpl
>> <http://localhost:7000/refsByType/0x6160e75d0>*
>>
>> 80
>>
>> *sun.security.rsa.RSAPublicKeyImpl
>> <http://localhost:7000/refsByType/0x6160e75d0>*
>>
>> 80
>>
>> *sun.security.x509.BasicConstraintsExtension
>> <http://localhost:7000/refsByType/0x616568910>*
>>
>> 72
>>
>> *sun.security.x509.BasicConstraintsExtension
>> <http://localhost:7000/refsByType/0x616568910>*
>>
>> 72
>>
>> *sun.security.x509.SubjectKeyIdentifierExtension
>> <http://localhost:7000/refsByType/0x616568890>*
>>
>> 65
>>
>> *sun.security.x509.SubjectKeyIdentifierExtension
>> <http://localhost:7000/refsByType/0x616568890>*
>>
>> 65
>>
>> *sun.security.x509.KeyUsageExtension
>> <http://localhost:7000/refsByType/0x615e96ec0>*
>>
>> 62
>>
>> *sun.security.x509.KeyUsageExtension
>> <http://localhost:7000/refsByType/0x615e96ec0>*
>>
>> 62
>>
>> *sun.security.ec.NamedCurve
>> <http://localhost:7000/refsByType/0x615e28ef0>*
>>
>> 46
>>
>> *sun.security.ec.NamedCurve
>> <http://localhost:7000/refsByType/0x615e28ef0>*
>>
>> 46
>>
>> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>>
>> 40
>>
>> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>>
>> 40
>>
>> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
>> <http://localhost:7000/refsByType/0x61637b1a0>*
>>
>> 34
>>
>> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
>> <http://localhost:7000/refsByType/0x61637b1a0>*
>>
>> 36
>>
>> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>>
>> 32
>>
>> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>>
>> 32
>>
>> *sun.security.x509.AuthorityKeyIdentifierExtension
>> <http://localhost:7000/refsByType/0x615f917c8>*
>>
>> 30
>>
>> *sun.security.x509.AuthorityKeyIdentifierExtension
>> <http://localhost:7000/refsByType/0x615f917c8>*
>>
>> 30
>>
>> *java.security.cert.PolicyQualifierInfo
>> <http://localhost:7000/refsByType/0x61672f090>*
>>
>> 28
>>
>> *java.security.cert.PolicyQualifierInfo
>> <http://localhost:7000/refsByType/0x61672f090>*
>>
>> 28
>>
>> *java.io.ByteArrayOutputStream
>> <http://localhost:7000/refsByType/0x6162c3f98>*
>>
>> 24
>>
>> *java.io.ByteArrayOutputStream
>> <http://localhost:7000/refsByType/0x6162c3f98>*
>>
>> 24
>>
>> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>>
>> 22
>>
>> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>>
>> 22
>>
>> *sun.security.x509.CRLDistributionPointsExtension
>> <http://localhost:7000/refsByType/0x616568810>*
>>
>> 17
>>
>> *sun.security.x509.CRLDistributionPointsExtension
>> <http://localhost:7000/refsByType/0x616568810>*
>>
>> 17
>>
>> *java.io.BufferedOutputStream
>> <http://localhost:7000/refsByType/0x61623db10>*
>>
>> 13
>>
>> *java.io.BufferedOutputStream
>> <http://localhost:7000/refsByType/0x61623db10>*
>>
>> 13
>>
>> *java.net.SocketOutputStream
>> <http://localhost:7000/refsByType/0x61665ded8>*
>>
>> 12
>>
>> *java.net.SocketOutputStream
>> <http://localhost:7000/refsByType/0x61665ded8>*
>>
>> 12
>>
>> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>>
>> 12
>>
>> *sun.security.x509.CertificatePoliciesExtension
>> <http://localhost:7000/refsByType/0x616568710>*
>>
>> 12
>>
>> *sun.security.x509.CertificatePoliciesExtension
>> <http://localhost:7000/refsByType/0x616568710>*
>>
>> 12
>>
>> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>>
>> 12
>>
>> *java.io.BufferedInputStream
>> <http://localhost:7000/refsByType/0x61669a630>*
>>
>> 10
>>
>> *java.io.BufferedInputStream
>> <http://localhost:7000/refsByType/0x61669a630>*
>>
>> 10
>>
>> *sun.security.provider.SecureRandom
>> <http://localhost:7000/refsByType/0x615f128c8>*
>>
>> 10
>>
>> *sun.security.provider.SecureRandom
>> <http://localhost:7000/refsByType/0x615f128c8>*
>>
>> 10
>>
>>
>>
>>
>>
>> *From:* moon soo Lee [mailto:moon@apache.org]
>> *Sent:* Thursday, February 25, 2016 10:56 PM
>> *To:* users@zeppelin.incubator.apache.org
>> *Subject:* Re: OOM error when run all paragraphs
>>
>>
>>
>> Hi Dafeng,
>>
>>
>>
>> Zeppelin keeps notebooks in memory.
>>
>> That may cause OOM when notebooks has a lot of con
>>
>>
Re: OOM error when run all paragraphs
Posted by moon soo Lee <mo...@apache.org>.
Thanks for creating an issue.
Let me look into it more.
Best,
moon
On Sun, Feb 28, 2016 at 10:14 PM Dafeng Wang <da...@microsoft.com> wrote:
> Thanks Moon for your effort to repro this bug, I had create a Jira
> https://issues.apache.org/jira/browse/ZEPPELIN-706 for it, please let me
> know if you need anything else.
>
>
>
> Regards,
>
> Dafeng
>
>
>
> *From:* moon soo Lee [mailto:moon@apache.org]
> *Sent:* Saturday, February 27, 2016 2:25 AM
>
>
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* Re: OOM error when run all paragraphs
>
>
>
> Thanks for sharing your use case and the memory usage table.
> I could able to reproduce the problem. Do you mind create an issue for it
> on our jira?
> I might have sometime next week to dig into this problem.
>
> Thanks,
> moon
>
>
>
> On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com> wrote:
>
> Hi Moon,
>
>
>
> Thanks for your reply, as for my case: my zeppelin server only have 1
> notebook with almost all queries are sparkSql query(30 paragraphs), the
> result limitation I set is 10000, all query reach the limitation of 10000.
>
> The strange thing is: run them one by one won’t cause the OOM error, if I
> click on run notebooks, it will OOM quickly.
>
>
>
> I check on the memory usage, the fastest increasing part is [B and [C,
> however, it just tells me we do have memory increase, doesn’t hint any
> other things, I put the table below for reference, the last one is the
> latest one, you can see only *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8> *got increased when I run
> it again and again, no obvious clue of memory leak.
>
>
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1090
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1091
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 115
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 121
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 34
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 36
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>
> 12
>
> *sun.security.x509.CertificatePoliciesExtension
> <http://localhost:7000/refsByType/0x616568710>*
>
> 12
>
> *sun.security.x509.CertificatePoliciesExtension
> <http://localhost:7000/refsByType/0x616568710>*
>
> 12
>
> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>
> 12
>
> *java.io.BufferedInputStream
> <http://localhost:7000/refsByType/0x61669a630>*
>
> 10
>
> *java.io.BufferedInputStream
> <http://localhost:7000/refsByType/0x61669a630>*
>
> 10
>
> *sun.security.provider.SecureRandom
> <http://localhost:7000/refsByType/0x615f128c8>*
>
> 10
>
> *sun.security.provider.SecureRandom
> <http://localhost:7000/refsByType/0x615f128c8>*
>
> 10
>
>
>
>
>
> *From:* moon soo Lee [mailto:moon@apache.org]
> *Sent:* Thursday, February 25, 2016 10:56 PM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* Re: OOM error when run all paragraphs
>
>
>
> Hi Dafeng,
>
>
>
> Zeppelin keeps notebooks in memory.
>
> That may cause OOM when notebooks has a lot of con
>
>
RE: OOM error when run all paragraphs
Posted by Dafeng Wang <da...@microsoft.com>.
Thanks Moon for your effort to repro this bug, I had create a Jira https://issues.apache.org/jira/browse/ZEPPELIN-706 for it, please let me know if you need anything else.
Regards,
Dafeng
From: moon soo Lee [mailto:moon@apache.org]
Sent: Saturday, February 27, 2016 2:25 AM
To: users@zeppelin.incubator.apache.org
Subject: Re: OOM error when run all paragraphs
Thanks for sharing your use case and the memory usage table.
I could able to reproduce the problem. Do you mind create an issue for it on our jira?
I might have sometime next week to dig into this problem.
Thanks,
moon
On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com>> wrote:
Hi Moon,
Thanks for your reply, as for my case: my zeppelin server only have 1 notebook with almost all queries are sparkSql query(30 paragraphs), the result limitation I set is 10000, all query reach the limitation of 10000.
The strange thing is: run them one by one won’t cause the OOM error, if I click on run notebooks, it will OOM quickly.
I check on the memory usage, the fastest increasing part is [B and [C, however, it just tells me we do have memory increase, doesn’t hint any other things, I put the table below for reference, the last one is the latest one, you can see only scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> got increased when I run it again and again, no obvious clue of memory leak.
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1090
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1091
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
[[B<http://localhost:7000/refsByType/0x616897178>
150
[[B<http://localhost:7000/refsByType/0x616897178>
150
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
115
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
121
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
34
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
36
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
From: moon soo Lee [mailto:moon@apache.org<ma...@apache.org>]
Sent: Thursday, February 25, 2016 10:56 PM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: Re: OOM error when run all paragraphs
Hi Dafeng,
Zeppelin keeps notebooks in memory.
That may cause OOM when notebooks has a lot of contents (code + result) inside. Unfortunately, unless modify Zeppelin to not keep notebooks in memory, there're no good way to reduce memory usage except for keep result small.
Most of notebooks i've seen is less than 2MB and i've seen people usually has ~300 notebooks in their Zeppelin instance.
Do you mind sharing your case? How big your notebook is and how many notebooks your Zeppelin instance have? That might help zeppelin developers to get idea how notebook need to be handled in ZeppelinServer.
Thanks,
moon
On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <da...@microsoft.com>> wrote:
It turns out zeppelin server itself oom since it cache too many results, are there any methods/work arounds/fixes to get this issue done?
Regards,
Dafeng
From: Dafeng Wang [mailto:dafewan@microsoft.com<ma...@microsoft.com>]
Sent: Thursday, February 25, 2016 10:31 AM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: OOM error when run all paragraphs
Hi,
Today when I try to run the zeppelin with a complicated notebook, if I run them one by one, it works well, however, when I click on running all paragraphs, it will throw OOM errors in a short time, I set zeppelin’s interpreter’s memory as 6G. any idea?
Regards,
Dafeng
Re: OOM error when run all paragraphs
Posted by moon soo Lee <mo...@apache.org>.
Thanks for sharing your use case and the memory usage table.
I could able to reproduce the problem. Do you mind create an issue for it
on our jira?
I might have sometime next week to dig into this problem.
Thanks,
moon
On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <da...@microsoft.com> wrote:
> Hi Moon,
>
>
>
> Thanks for your reply, as for my case: my zeppelin server only have 1
> notebook with almost all queries are sparkSql query(30 paragraphs), the
> result limitation I set is 10000, all query reach the limitation of 10000.
>
> The strange thing is: run them one by one won’t cause the OOM error, if I
> click on run notebooks, it will OOM quickly.
>
>
>
> I check on the memory usage, the fastest increasing part is [B and [C,
> however, it just tells me we do have memory increase, doesn’t hint any
> other things, I put the table below for reference, the last one is the
> latest one, you can see only *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8> *got increased when I run
> it again and again, no obvious clue of memory leak.
>
>
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.FileZipArchive$FileEntry$1
> <http://localhost:7000/refsByType/0x616022ae8>*
>
> 224886
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *scala.reflect.io.ZipArchive$DirEntry
> <http://localhost:7000/refsByType/0x616022ca8>*
>
> 10123
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1090
>
> *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>*
>
> 1091
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>*
>
> 960
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *org.apache.derby.iapi.services.io.FormatIdInputStream
> <http://localhost:7000/refsByType/0x61b38f2a0>*
>
> 884
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *sun.security.util.DerInputBuffer
> <http://localhost:7000/refsByType/0x61684fc40>*
>
> 745
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *org.apache.derby.iapi.services.io.FormatIdOutputStream
> <http://localhost:7000/refsByType/0x61b3f5f20>*
>
> 479
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *sun.security.util.ObjectIdentifier
> <http://localhost:7000/refsByType/0x615d02320>*
>
> 466
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayInputStream
> <http://localhost:7000/refsByType/0x61b215910>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.ArrayOutputStream
> <http://localhost:7000/refsByType/0x61b3f6000>*
>
> 442
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *org.apache.derby.iapi.services.io.FormatableBitSet
> <http://localhost:7000/refsByType/0x617ae17e8>*
>
> 364
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *scala.reflect.internal.pickling.UnPickler$Scan
> <http://localhost:7000/refsByType/0x6166d6e20>*
>
> 282
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *org.apache.derby.impl.store.raw.data.StoredPage
> <http://localhost:7000/refsByType/0x61ba57d88>*
>
> 279
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *java.util.jar.JarFile$JarFileEntry
> <http://localhost:7000/refsByType/0x615e24618>*
>
> 276
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *sun.security.x509.X509CertImpl
> <http://localhost:7000/refsByType/0x616083cf0>*
>
> 166
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.apache.derby.impl.store.raw.data.AllocPage
> <http://localhost:7000/refsByType/0x61b15ef28>*
>
> 162
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *org.spark-project.jetty.io.BufferCache$CachedBuffer
> <http://localhost:7000/refsByType/0x616325310>*
>
> 155
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *[[B <http://localhost:7000/refsByType/0x616897178>*
>
> 150
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.spark-project.jetty.io.ByteArrayBuffer
> <http://localhost:7000/refsByType/0x616309b60>*
>
> 138
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *org.apache.thrift.protocol.TBinaryProtocol
> <http://localhost:7000/refsByType/0x615f91410>*
>
> 128
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 115
>
> *scala.reflect.io.VirtualFile
> <http://localhost:7000/refsByType/0x616022bc8>*
>
> 121
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *org.apache.derby.impl.store.raw.data.RAFContainer4
> <http://localhost:7000/refsByType/0x61b974dc0>*
>
> 100
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *sun.security.x509.KeyIdentifier
> <http://localhost:7000/refsByType/0x616856000>*
>
> 95
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>*
>
> 94
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.util.Cache$EqualByteArray
> <http://localhost:7000/refsByType/0x61688c258>*
>
> 84
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>*
>
> 83
>
> *sun.security.x509.X509CertInfo
> <http://localhost:7000/refsByType/0x6163a8398>*
>
> 83
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.rsa.RSAPublicKeyImpl
> <http://localhost:7000/refsByType/0x6160e75d0>*
>
> 80
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.BasicConstraintsExtension
> <http://localhost:7000/refsByType/0x616568910>*
>
> 72
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.SubjectKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x616568890>*
>
> 65
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.x509.KeyUsageExtension
> <http://localhost:7000/refsByType/0x615e96ec0>*
>
> 62
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>*
>
> 46
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>*
>
> 40
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 34
>
> *scala.tools.nsc.symtab.classfile.Pickler$Pickle
> <http://localhost:7000/refsByType/0x61637b1a0>*
>
> 36
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>*
>
> 32
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *sun.security.x509.AuthorityKeyIdentifierExtension
> <http://localhost:7000/refsByType/0x615f917c8>*
>
> 30
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.security.cert.PolicyQualifierInfo
> <http://localhost:7000/refsByType/0x61672f090>*
>
> 28
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.io.ByteArrayOutputStream
> <http://localhost:7000/refsByType/0x6162c3f98>*
>
> 24
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>*
>
> 22
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *sun.security.x509.CRLDistributionPointsExtension
> <http://localhost:7000/refsByType/0x616568810>*
>
> 17
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.io.BufferedOutputStream
> <http://localhost:7000/refsByType/0x61623db10>*
>
> 13
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *java.net.SocketOutputStream
> <http://localhost:7000/refsByType/0x61665ded8>*
>
> 12
>
> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>
> 12
>
> *sun.security.x509.CertificatePoliciesExtension
> <http://localhost:7000/refsByType/0x616568710>*
>
> 12
>
> *sun.security.x509.CertificatePoliciesExtension
> <http://localhost:7000/refsByType/0x616568710>*
>
> 12
>
> *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>*
>
> 12
>
> *java.io.BufferedInputStream
> <http://localhost:7000/refsByType/0x61669a630>*
>
> 10
>
> *java.io.BufferedInputStream
> <http://localhost:7000/refsByType/0x61669a630>*
>
> 10
>
> *sun.security.provider.SecureRandom
> <http://localhost:7000/refsByType/0x615f128c8>*
>
> 10
>
> *sun.security.provider.SecureRandom
> <http://localhost:7000/refsByType/0x615f128c8>*
>
> 10
>
>
>
>
>
> *From:* moon soo Lee [mailto:moon@apache.org]
> *Sent:* Thursday, February 25, 2016 10:56 PM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* Re: OOM error when run all paragraphs
>
>
>
> Hi Dafeng,
>
>
>
> Zeppelin keeps notebooks in memory.
>
> That may cause OOM when notebooks has a lot of contents (code + result)
> inside. Unfortunately, unless modify Zeppelin to not keep notebooks in
> memory, there're no good way to reduce memory usage except for keep result
> small.
>
>
>
> Most of notebooks i've seen is less than 2MB and i've seen people usually
> has ~300 notebooks in their Zeppelin instance.
>
>
>
> Do you mind sharing your case? How big your notebook is and how many
> notebooks your Zeppelin instance have? That might help zeppelin developers
> to get idea how notebook need to be handled in ZeppelinServer.
>
>
>
> Thanks,
>
> moon
>
>
>
> On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <da...@microsoft.com> wrote:
>
> It turns out zeppelin server itself oom since it cache too many results,
> are there any methods/work arounds/fixes to get this issue done?
>
>
>
> Regards,
>
> Dafeng
>
>
>
> *From:* Dafeng Wang [mailto:dafewan@microsoft.com]
> *Sent:* Thursday, February 25, 2016 10:31 AM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* OOM error when run all paragraphs
>
>
>
> Hi,
>
> Today when I try to run the zeppelin with a complicated notebook, if I run
> them one by one, it works well, however, when I click on running all
> paragraphs, it will throw OOM errors in a short time, I set zeppelin’s
> interpreter’s memory as 6G. any idea?
>
> Regards,
>
> Dafeng
>
>
RE: OOM error when run all paragraphs
Posted by Dafeng Wang <da...@microsoft.com>.
Hi Moon,
Thanks for your reply, as for my case: my zeppelin server only have 1 notebook with almost all queries are sparkSql query(30 paragraphs), the result limitation I set is 10000, all query reach the limitation of 10000.
The strange thing is: run them one by one won’t cause the OOM error, if I click on run notebooks, it will OOM quickly.
I check on the memory usage, the fastest increasing part is [B and [C, however, it just tells me we do have memory increase, doesn’t hint any other things, I put the table below for reference, the last one is the latest one, you can see only scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> got increased when I run it again and again, no obvious clue of memory leak.
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8>
224886
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8>
10123
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1090
sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160>
1091
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20>
960
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0>
884
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40>
745
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20>
479
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320>
466
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000>
442
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8>
364
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20>
282
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88>
279
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618>
276
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0>
166
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28>
162
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310>
155
[[B<http://localhost:7000/refsByType/0x616897178>
150
[[B<http://localhost:7000/refsByType/0x616897178>
150
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60>
138
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410>
128
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
115
scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8>
121
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0>
100
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000>
95
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
java.lang.Class<http://localhost:7000/refsByType/0x615e60220>
94
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258>
84
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0>
83
sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398>
83
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0>
80
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910>
72
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890>
65
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0>
62
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0>
46
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0>
40
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
34
scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0>
36
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0>
32
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8>
30
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090>
28
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98>
24
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838>
22
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810>
17
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10>
13
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710>
12
sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0>
12
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8>
10
From: moon soo Lee [mailto:moon@apache.org]
Sent: Thursday, February 25, 2016 10:56 PM
To: users@zeppelin.incubator.apache.org
Subject: Re: OOM error when run all paragraphs
Hi Dafeng,
Zeppelin keeps notebooks in memory.
That may cause OOM when notebooks has a lot of contents (code + result) inside. Unfortunately, unless modify Zeppelin to not keep notebooks in memory, there're no good way to reduce memory usage except for keep result small.
Most of notebooks i've seen is less than 2MB and i've seen people usually has ~300 notebooks in their Zeppelin instance.
Do you mind sharing your case? How big your notebook is and how many notebooks your Zeppelin instance have? That might help zeppelin developers to get idea how notebook need to be handled in ZeppelinServer.
Thanks,
moon
On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <da...@microsoft.com>> wrote:
It turns out zeppelin server itself oom since it cache too many results, are there any methods/work arounds/fixes to get this issue done?
Regards,
Dafeng
From: Dafeng Wang [mailto:dafewan@microsoft.com<ma...@microsoft.com>]
Sent: Thursday, February 25, 2016 10:31 AM
To: users@zeppelin.incubator.apache.org<ma...@zeppelin.incubator.apache.org>
Subject: OOM error when run all paragraphs
Hi,
Today when I try to run the zeppelin with a complicated notebook, if I run them one by one, it works well, however, when I click on running all paragraphs, it will throw OOM errors in a short time, I set zeppelin’s interpreter’s memory as 6G. any idea?
Regards,
Dafeng
Re: OOM error when run all paragraphs
Posted by moon soo Lee <mo...@apache.org>.
Hi Dafeng,
Zeppelin keeps notebooks in memory.
That may cause OOM when notebooks has a lot of contents (code + result)
inside. Unfortunately, unless modify Zeppelin to not keep notebooks in
memory, there're no good way to reduce memory usage except for keep result
small.
Most of notebooks i've seen is less than 2MB and i've seen people usually
has ~300 notebooks in their Zeppelin instance.
Do you mind sharing your case? How big your notebook is and how many
notebooks your Zeppelin instance have? That might help zeppelin developers
to get idea how notebook need to be handled in ZeppelinServer.
Thanks,
moon
On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <da...@microsoft.com> wrote:
> It turns out zeppelin server itself oom since it cache too many results,
> are there any methods/work arounds/fixes to get this issue done?
>
>
>
> Regards,
>
> Dafeng
>
>
>
> *From:* Dafeng Wang [mailto:dafewan@microsoft.com]
> *Sent:* Thursday, February 25, 2016 10:31 AM
> *To:* users@zeppelin.incubator.apache.org
> *Subject:* OOM error when run all paragraphs
>
>
>
> Hi,
>
> Today when I try to run the zeppelin with a complicated notebook, if I run
> them one by one, it works well, however, when I click on running all
> paragraphs, it will throw OOM errors in a short time, I set zeppelin’s
> interpreter’s memory as 6G. any idea?
>
> Regards,
>
> Dafeng
>
>
RE: OOM error when run all paragraphs
Posted by Dafeng Wang <da...@microsoft.com>.
It turns out zeppelin server itself oom since it cache too many results, are there any methods/work arounds/fixes to get this issue done?
Regards,
Dafeng
From: Dafeng Wang [mailto:dafewan@microsoft.com]
Sent: Thursday, February 25, 2016 10:31 AM
To: users@zeppelin.incubator.apache.org
Subject: OOM error when run all paragraphs
Hi,
Today when I try to run the zeppelin with a complicated notebook, if I run them one by one, it works well, however, when I click on running all paragraphs, it will throw OOM errors in a short time, I set zeppelin’s interpreter’s memory as 6G. any idea?
Regards,
Dafeng