You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sreetej Lakkam (JIRA)" <ji...@apache.org> on 2016/06/09 13:03:21 UTC
[jira] [Created] (SPARK-15843) Spark RAM issue
Sreetej Lakkam created SPARK-15843:
--------------------------------------
Summary: Spark RAM issue
Key: SPARK-15843
URL: https://issues.apache.org/jira/browse/SPARK-15843
Project: Spark
Issue Type: Question
Components: Spark Shell
Affects Versions: 1.6.1
Environment:
RASPBIAN JESSIE
Full desktop image based on Debian Jessie
Reporter: Sreetej Lakkam
Trying to run Spark 1.6.1 on Hadoop 2.7.1 over Raspberry Pi - 3. On submitting spark-shell
sudo /opt/spark-1.6.1-bin-hadoop2.6/bin/spark-shell
produces an error
Java HotSpot(TM) Client VM warning: INFO: os::commit_memory(0x49540000, 715915264, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/pi/hs_err_pid2179.log
Find the log file hs_err_pid2179.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2627), pid=2179, tid=1983399008
#
# JRE version: (8.0_65-b17) (build )
# Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode, sharing linux-arm )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x76207400): JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)]
Stack: [0x76335000,0x76385000]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x76207400 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
00008000-00009000 r-xp 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java
00010000-00011000 rw-p 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java
003e5000-00406000 rw-p 00000000 00:00 0 [heap]
33dff000-33eaa000 rw-p 00000000 00:00 0
33eaa000-33fff000 ---p 00000000 00:00 0
33fff000-49540000 rw-p 00000000 00:00 0
740c3000-740c4000 rw-p 00000000 00:00 0
740c4000-74143000 ---p 00000000 00:00 0
74143000-7416b000 rwxp 00000000 00:00 0
7416b000-76143000 ---p 00000000 00:00 0
76143000-7615a000 r-xp 00000000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
7615a000-76161000 ---p 00017000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
76161000-76162000 rw-p 00016000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so
76162000-7616d000 r-xp 00000000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
7616d000-7617c000 ---p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
7617c000-7617d000 r--p 0000a000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
7617d000-7617e000 rw-p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
7617e000-76187000 r-xp 00000000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
76187000-76196000 ---p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
76196000-76197000 r--p 00008000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
76197000-76198000 rw-p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
76198000-761a9000 r-xp 00000000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
761a9000-761b8000 ---p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
761b8000-761b9000 r--p 00010000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
761b9000-761ba000 rw-p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so
761ba000-761bc000 rw-p 00000000 00:00 0
761bc000-761c3000 r-xp 00000000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
761c3000-761d2000 ---p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
761d2000-761d3000 r--p 00006000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
761d3000-761d4000 rw-p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
761d4000-761f7000 r-xp 00000000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
761f7000-761ff000 ---p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
761ff000-76200000 rw-p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so
76200000-76221000 rw-p 00000000 00:00 0
76221000-76300000 ---p 00000000 00:00 0
7631e000-76324000 r-xp 00000000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
76324000-76333000 ---p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
76333000-76334000 r--p 00005000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
76334000-76335000 rw-p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so
76335000-76338000 ---p 00000000 00:00 0
76338000-76385000 rwxp 00000000 00:00 0 [stack:2200]
76385000-763ee000 r-xp 00000000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
763ee000-763fe000 ---p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
763fe000-763ff000 r--p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
763ff000-76400000 rw-p 0006a000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so
76400000-76983000 r-xp 00000000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
76983000-7698b000 ---p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
7698b000-769b7000 rw-p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so
769b7000-76dda000 rw-p 00000000 00:00 0
76dda000-76df7000 r-xp 00000000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
76df7000-76e06000 ---p 0001d000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
76e06000-76e07000 rw-p 0001c000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1
76e07000-76f32000 r-xp 00000000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
76f32000-76f42000 ---p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
76f42000-76f44000 r--p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
76f44000-76f45000 rw-p 0012d000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so
76f45000-76f48000 rw-p 00000000 00:00 0
76f48000-76f4a000 r-xp 00000000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
76f4a000-76f59000 ---p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
76f59000-76f5a000 r--p 00001000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
76f5a000-76f5b000 rw-p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so
76f5b000-76f6f000 r-xp 00000000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
76f6f000-76f77000 ---p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
76f77000-76f78000 rw-p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so
76f78000-76f8c000 r-xp 00000000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
76f8c000-76f9c000 ---p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
76f9c000-76f9d000 r--p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
76f9d000-76f9e000 rw-p 00015000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so
76f9e000-76fa0000 rw-p 00000000 00:00 0
76fa1000-76faa000 r-xp 00000000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
76faa000-76fb2000 ---p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
76fb2000-76fb3000 rw-p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so
76fb3000-76fb8000 r-xp 00000000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
76fb8000-76fc7000 ---p 00005000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
76fc7000-76fc8000 rw-p 00004000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so
76fc8000-76fe8000 r-xp 00000000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
76fe9000-76ff1000 rw-s 00000000 b3:02 409006 /tmp/hsperfdata_root/2179
76ff1000-76ff2000 r--p 00000000 00:00 0
76ff2000-76ff5000 rw-p 00000000 00:00 0
76ff5000-76ff7000 rw-p 00000000 00:00 0
76ff7000-76ff8000 r--p 0001f000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
76ff8000-76ff9000 rw-p 00020000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so
7ec8f000-7ecb0000 rwxp 00000000 00:00 0 [stack]
7ed85000-7ed86000 r-xp 00000000 00:00 0 [sigpage]
7ed86000-7ed87000 r--p 00000000 00:00 0 [vvar]
7ed87000-7ed88000 r-xp 00000000 00:00 0 [vdso]
ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]
VM Arguments:
jvm_args: -Dscala.usejavacp=true -Xms1g -Xmx1g
java_command: org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name Spark shell spark-shell
java_class_path (initial): /opt/spark-1.6.1-bin-hadoop2.6/conf/:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
USERNAME=root
SHELL=/bin/bash
DISPLAY=:0.0
Signal Handlers:
SIGSEGV: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x3fd650], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
--------------- S Y S T E M ---------------
OS:PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
uname:Linux 4.1.18-v7+ #846 SMP Thu Feb 25 14:22:53 GMT 2016 armv7l
libc:glibc 2.19 NPTL 2.19
rlimit: STACK 8192k, CORE 0k, NPROC 7719, NOFILE 65536, AS infinity
load average:1.36 0.61 0.23
/proc/meminfo:
MemTotal: 996776 kB
MemFree: 339060 kB
MemAvailable: 531380 kB
Buffers: 19752 kB
Cached: 197208 kB
SwapCached: 0 kB
Active: 500964 kB
Inactive: 119008 kB
Active(anon): 403400 kB
Inactive(anon): 7856 kB
Active(file): 97564 kB
Inactive(file): 111152 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 102396 kB
SwapFree: 102396 kB
Dirty: 156 kB
Writeback: 0 kB
AnonPages: 402992 kB
Mapped: 77148 kB
Shmem: 8316 kB
Slab: 17608 kB
SReclaimable: 8168 kB
SUnreclaim: 9440 kB
KernelStack: 4304 kB
PageTables: 3788 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 600784 kB
Committed_AS: 1410392 kB
VmallocTotal: 1056768 kB
VmallocUsed: 5176 kB
VmallocChunk: 806504 kB
CmaTotal: 8192 kB
CmaFree: 3732 kB
CPU:total 4 (ARMv7), vfp
/proc/cpuinfo:
processor : 0
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 1
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 2
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
Hardware : BCM2709
Revision : a02082
Serial : 0000000010852207
Memory: 4k page, physical 996776k(339060k free), swap 102396k(102396k free)
vm_info: Java HotSpot(TM) Client VM (25.65-b01) for linux-arm-vfp-hflt JRE (1.8.0_65-b17), built on Oct 6 2015 16:19:04 by "java_re" with gcc 4.7.2 20120910 (prerelease)
time: Wed Jun 8 22:49:10 2016
elapsed time: 0 seconds (0d 0h 0m 0s)
Could you please help me resolve the issue??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org