You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/06/09 13:32:21 UTC
[jira] [Resolved] (SPARK-15843) Spark RAM issue
[ https://issues.apache.org/jira/browse/SPARK-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-15843.
-------------------------------
Resolution: Invalid
Target Version/s: (was: 1.6.1)
Please read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark and ask questions at user@.
This means your JVM ran out of OS memory, probably because you asked for more heap than the OS can allocate and perhaps swap is off. It's not a Spark issue.
> Spark RAM issue
> ---------------
>
> Key: SPARK-15843
> URL: https://issues.apache.org/jira/browse/SPARK-15843
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
> Affects Versions: 1.6.1
> Environment: RASPBIAN JESSIE
> Full desktop image based on Debian Jessie
> Reporter: Sreetej Lakkam
> Labels: beginner, newbie
>
> Trying to run Spark 1.6.1 on Hadoop 2.7.1 over Raspberry Pi - 3. On submitting spark-shell
> sudo /opt/spark-1.6.1-bin-hadoop2.6/bin/spark-shell
> produces an error
> Java HotSpot(TM) Client VM warning: INFO: os::commit_memory(0x49540000, 715915264, 0) failed; error='Cannot allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory.
> # An error report file with more information is saved as:
> # /home/pi/hs_err_pid2179.log
> Find the log file hs_err_pid2179.log
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # In 32 bit mode, the process size limit was hit
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Use 64 bit Java on a 64 bit OS
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2627), pid=2179, tid=1983399008
> #
> # JRE version: (8.0_65-b17) (build )
> # Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode, sharing linux-arm )
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> --------------- T H R E A D ---------------
> Current thread (0x76207400): JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)]
> Stack: [0x76335000,0x76385000]
> --------------- P R O C E S S ---------------
> Java Threads: ( => current thread )
> Other Threads:
> =>0x76207400 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)]
> VM state:not at safepoint (not fully initialized)
> VM Mutex/Monitor currently owned by a thread: None
> GC Heap History (0 events):
> No events
> Deoptimization events (0 events):
> No events
> Internal exceptions (0 events):
> No events
> Events (0 events):
> No events
> Dynamic libraries:
> 00008000-00009000 r-xp 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java
> 00010000-00011000 rw-p 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java
> 003e5000-00406000 rw-p 00000000 00:00 0 [heap]
> 33dff000-33eaa000 rw-p 00000000 00:00 0
> 33eaa000-33fff000 ---p 00000000 00:00 0
> 33fff000-49540000 rw-p 00000000 00:00 0
> 740c3000-740c4000 rw-p 00000000 00:00 0
> 740c4000-74143000 ---p 00000000 00:00 0
> 74143000-7416b000 rwxp 00000000 00:00 0
> 7416b000-76143000 ---p 00000000 00:00 0
> 76143000-7615a000 r-xp 00000000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
> 7615a000-76161000 ---p 00017000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
> 76161000-76162000 rw-p 00016000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
> 76162000-7616d000 r-xp 00000000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
> 7616d000-7617c000 ---p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
> 7617c000-7617d000 r--p 0000a000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
> 7617d000-7617e000 rw-p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
> 7617e000-76187000 r-xp 00000000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
> 76187000-76196000 ---p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
> 76196000-76197000 r--p 00008000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
> 76197000-76198000 rw-p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
> 76198000-761a9000 r-xp 00000000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
> 761a9000-761b8000 ---p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
> 761b8000-761b9000 r--p 00010000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
> 761b9000-761ba000 rw-p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
> 761ba000-761bc000 rw-p 00000000 00:00 0
> 761bc000-761c3000 r-xp 00000000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
> 761c3000-761d2000 ---p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
> 761d2000-761d3000 r--p 00006000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
> 761d3000-761d4000 rw-p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
> 761d4000-761f7000 r-xp 00000000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
> 761f7000-761ff000 ---p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
> 761ff000-76200000 rw-p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
> 76200000-76221000 rw-p 00000000 00:00 0
> 76221000-76300000 ---p 00000000 00:00 0
> 7631e000-76324000 r-xp 00000000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
> 76324000-76333000 ---p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
> 76333000-76334000 r--p 00005000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
> 76334000-76335000 rw-p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
> 76335000-76338000 ---p 00000000 00:00 0
> 76338000-76385000 rwxp 00000000 00:00 0 [stack:2200]
> 76385000-763ee000 r-xp 00000000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
> 763ee000-763fe000 ---p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
> 763fe000-763ff000 r--p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
> 763ff000-76400000 rw-p 0006a000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
> 76400000-76983000 r-xp 00000000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
> 76983000-7698b000 ---p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
> 7698b000-769b7000 rw-p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
> 769b7000-76dda000 rw-p 00000000 00:00 0
> 76dda000-76df7000 r-xp 00000000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
> 76df7000-76e06000 ---p 0001d000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
> 76e06000-76e07000 rw-p 0001c000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
> 76e07000-76f32000 r-xp 00000000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
> 76f32000-76f42000 ---p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
> 76f42000-76f44000 r--p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
> 76f44000-76f45000 rw-p 0012d000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
> 76f45000-76f48000 rw-p 00000000 00:00 0
> 76f48000-76f4a000 r-xp 00000000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
> 76f4a000-76f59000 ---p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
> 76f59000-76f5a000 r--p 00001000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
> 76f5a000-76f5b000 rw-p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
> 76f5b000-76f6f000 r-xp 00000000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
> 76f6f000-76f77000 ---p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
> 76f77000-76f78000 rw-p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
> 76f78000-76f8c000 r-xp 00000000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
> 76f8c000-76f9c000 ---p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
> 76f9c000-76f9d000 r--p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
> 76f9d000-76f9e000 rw-p 00015000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
> 76f9e000-76fa0000 rw-p 00000000 00:00 0
> 76fa1000-76faa000 r-xp 00000000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
> 76faa000-76fb2000 ---p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
> 76fb2000-76fb3000 rw-p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
> 76fb3000-76fb8000 r-xp 00000000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
> 76fb8000-76fc7000 ---p 00005000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
> 76fc7000-76fc8000 rw-p 00004000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
> 76fc8000-76fe8000 r-xp 00000000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
> 76fe9000-76ff1000 rw-s 00000000 b3:02 409006 /tmp/hsperfdata_root/2179
> 76ff1000-76ff2000 r--p 00000000 00:00 0
> 76ff2000-76ff5000 rw-p 00000000 00:00 0
> 76ff5000-76ff7000 rw-p 00000000 00:00 0
> 76ff7000-76ff8000 r--p 0001f000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
> 76ff8000-76ff9000 rw-p 00020000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
> 7ec8f000-7ecb0000 rwxp 00000000 00:00 0 [stack]
> 7ed85000-7ed86000 r-xp 00000000 00:00 0 [sigpage]
> 7ed86000-7ed87000 r--p 00000000 00:00 0 [vvar]
> 7ed87000-7ed88000 r-xp 00000000 00:00 0 [vdso]
> ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]
> VM Arguments:
> jvm_args: -Dscala.usejavacp=true -Xms1g -Xmx1g
> java_command: org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name Spark shell spark-shell
> java_class_path (initial): /opt/spark-1.6.1-bin-hadoop2.6/conf/:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
> Launcher Type: SUN_STANDARD
> Environment Variables:
> JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> USERNAME=root
> SHELL=/bin/bash
> DISPLAY=:0.0
> Signal Handlers:
> SIGSEGV: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGBUS: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGFPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGPIPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGXFSZ: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGILL: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
> SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
> SIGUSR2: [libjvm.so+0x3fd650], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
> SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
> SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
> SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
> SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
> --------------- S Y S T E M ---------------
> OS:PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
> NAME="Raspbian GNU/Linux"
> VERSION_ID="8"
> VERSION="8 (jessie)"
> ID=raspbian
> ID_LIKE=debian
> HOME_URL="http://www.raspbian.org/"
> SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
> BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
> uname:Linux 4.1.18-v7+ #846 SMP Thu Feb 25 14:22:53 GMT 2016 armv7l
> libc:glibc 2.19 NPTL 2.19
> rlimit: STACK 8192k, CORE 0k, NPROC 7719, NOFILE 65536, AS infinity
> load average:1.36 0.61 0.23
> /proc/meminfo:
> MemTotal: 996776 kB
> MemFree: 339060 kB
> MemAvailable: 531380 kB
> Buffers: 19752 kB
> Cached: 197208 kB
> SwapCached: 0 kB
> Active: 500964 kB
> Inactive: 119008 kB
> Active(anon): 403400 kB
> Inactive(anon): 7856 kB
> Active(file): 97564 kB
> Inactive(file): 111152 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 102396 kB
> SwapFree: 102396 kB
> Dirty: 156 kB
> Writeback: 0 kB
> AnonPages: 402992 kB
> Mapped: 77148 kB
> Shmem: 8316 kB
> Slab: 17608 kB
> SReclaimable: 8168 kB
> SUnreclaim: 9440 kB
> KernelStack: 4304 kB
> PageTables: 3788 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 600784 kB
> Committed_AS: 1410392 kB
> VmallocTotal: 1056768 kB
> VmallocUsed: 5176 kB
> VmallocChunk: 806504 kB
> CmaTotal: 8192 kB
> CmaFree: 3732 kB
> CPU:total 4 (ARMv7), vfp
> /proc/cpuinfo:
> processor : 0
> model name : ARMv7 Processor rev 4 (v7l)
> BogoMIPS : 76.80
> Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
> CPU implementer : 0x41
> CPU architecture: 7
> CPU variant : 0x0
> CPU part : 0xd03
> CPU revision : 4
> processor : 1
> model name : ARMv7 Processor rev 4 (v7l)
> BogoMIPS : 76.80
> Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
> CPU implementer : 0x41
> CPU architecture: 7
> CPU variant : 0x0
> CPU part : 0xd03
> CPU revision : 4
> processor : 2
> model name : ARMv7 Processor rev 4 (v7l)
> BogoMIPS : 76.80
> Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
> CPU implementer : 0x41
> CPU architecture: 7
> CPU variant : 0x0
> CPU part : 0xd03
> CPU revision : 4
> processor : 3
> model name : ARMv7 Processor rev 4 (v7l)
> BogoMIPS : 76.80
> Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
> CPU implementer : 0x41
> CPU architecture: 7
> CPU variant : 0x0
> CPU part : 0xd03
> CPU revision : 4
> Hardware : BCM2709
> Revision : a02082
> Serial : 0000000010852207
> Memory: 4k page, physical 996776k(339060k free), swap 102396k(102396k free)
> vm_info: Java HotSpot(TM) Client VM (25.65-b01) for linux-arm-vfp-hflt JRE (1.8.0_65-b17), built on Oct 6 2015 16:19:04 by "java_re" with gcc 4.7.2 20120910 (prerelease)
> time: Wed Jun 8 22:49:10 2016
> elapsed time: 0 seconds (0d 0h 0m 0s)
> Could you please help me resolve the issue??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org