You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by John Simon <jo...@tapjoy.com> on 2016/06/09 18:09:47 UTC

Spark Streaming getting slower

Hi,

I'm running Spark Streaming with Kafka Direct Stream, batch interval
is 10 seconds.
After running about 72 hours, the batch processing time almost doubles.
I didn't find anything wrong on JVM GC logs, but I did find that
broadcast variable reading time increasing, like this:

initially:

```
16/06/08 18:17:02 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 0 took 223 ms
16/06/08 18:17:08 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 2 took 73 ms
16/06/08 18:17:10 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 3 took 13 ms
16/06/08 18:17:11 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 4 took 9 ms
16/06/08 18:17:12 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 5 took 8 ms
16/06/08 18:17:14 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 6 took 13 ms
16/06/08 18:17:15 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 8 took 7 ms
16/06/08 18:17:16 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 10 took 8 ms
16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 12 took 6 ms
16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 13 took 7 ms
```

after 23 hours:

```
16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282938 took 18 ms
16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282939 took 20 ms
16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282940 took 20 ms
16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282941 took 14 ms
16/06/09 17:23:44 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282942 took 17 ms
16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282943 took 18 ms
16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282944 took 14 ms
16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282945 took 18 ms
16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282946 took 14 ms
16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 282947 took 18 ms
```

FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client mode.

application environment follows:

```
Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre
Java Version 1.7.0_101 (Oracle Corporation)
Scala Version version 2.10.5

spark.app.id application_1463430559850_0056
spark.cleaner.ttl 60
spark.default.parallelism 96
spark.driver.appUIAddress http://172.16.4.168:4041
spark.driver.extraClassPath
/etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
spark.driver.extraJavaOptions
-Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
-XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p'
spark.driver.extraLibraryPath
/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.driver.host 172.16.4.168
spark.driver.port 42142
spark.dynamicAllocation.enabled false
spark.eventLog.compress true
spark.eventLog.dir hdfs:///var/log/spark/apps
spark.eventLog.enabled false
spark.executor.cores 8
spark.executor.extraClassPath
/etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70
-XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraLibraryPath
/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.executor.id driver
spark.executor.instances 24
spark.executor.memory 6G
spark.externalBlockStore.folderName spark-9ede6685-a7f9-4d65-bfb4-0920e8c5ba25
spark.history.fs.cleaner.enabled true
spark.history.fs.cleaner.maxAge 2d
spark.history.fs.logDirectory hdfs:///var/log/spark/apps
spark.history.ui.port 18080
spark.master yarn-client
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS
ip-172-16-4-168.ec2.internal
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES
http://ip-172-16-4-168.ec2.internal:20888/proxy/application_1463430559850_0056
spark.scheduler.mode FAIR
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.shuffle.service.enabled false
spark.streaming.backpressure.enabled true
spark.streaming.concurrentJobs 2
spark.streaming.kafka.maxRatePerPartition 312
spark.streaming.ui.retainedBatches 1000
spark.submit.deployMode client
spark.ui.filters org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
spark.ui.retainedJobs 1000
spark.ui.retainedStages 1000
spark.yarn.executor.memoryOverhead 4096
spark.yarn.historyServer.address ip-172-16-4-168.ec2.internal:18080

SPARK_SUBMIT true
SPARK_YARN_MODE true
awt.toolkit sun.awt.X11.XToolkit
file.encoding UTF-8
file.encoding.pkg sun.io
file.separator /
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.printerjob sun.print.PSPrinterJob
java.class.version 51.0
java.endorsed.dirs
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/endorsed
java.ext.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre
java.io.tmpdir /tmp
java.library.path
/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.7.0_101-mockbuild_2016_04_26_18_10-b00
java.specification.name Java Platform API Specification
java.specification.vendor Oracle Corporation
java.specification.version 1.7
java.vendor Oracle Corporation
java.vendor.url http://java.oracle.com/
java.vendor.url.bug http://bugreport.sun.com/bugreport/
java.version 1.7.0_101
java.vm.info mixed mode
java.vm.name OpenJDK 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Oracle Corporation
java.vm.specification.version 1.7
java.vm.vendor Oracle Corporation
java.vm.version 24.95-b01
line.separator
log4j.configuration file:///etc/spark/conf/log4j.properties
os.arch amd64
os.name Linux
os.version 4.4.5-15.26.amzn1.x86_64
path.separator :
sun.arch.data.model 64
sun.boot.class.path
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/classes
sun.boot.library.path
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/amd64
sun.cpu.endian little
sun.cpu.isalist
sun.io.unicode.encoding UnicodeLittle
sun.java.launcher SUN_STANDARD
sun.jnu.encoding UTF-8
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.nio.ch.bugLevel
sun.os.patch.level unknown
user.country US
user.dir /home/hadoop
user.home /home/hadoop
user.language en
user.name hadoop
user.timezone UTC
```


--
John Simon

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark Streaming getting slower

Posted by John Simon <jo...@tapjoy.com>.
Sorry, forgot to mention that I don't use broadcast variables. That's
why I'm puzzled here.
--
John Simon


On Thu, Jun 9, 2016 at 11:09 AM, John Simon <jo...@tapjoy.com> wrote:
> Hi,
>
> I'm running Spark Streaming with Kafka Direct Stream, batch interval
> is 10 seconds.
> After running about 72 hours, the batch processing time almost doubles.
> I didn't find anything wrong on JVM GC logs, but I did find that
> broadcast variable reading time increasing, like this:
>
> initially:
>
> ```
> 16/06/08 18:17:02 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 0 took 223 ms
> 16/06/08 18:17:08 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 2 took 73 ms
> 16/06/08 18:17:10 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 3 took 13 ms
> 16/06/08 18:17:11 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 4 took 9 ms
> 16/06/08 18:17:12 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 5 took 8 ms
> 16/06/08 18:17:14 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 6 took 13 ms
> 16/06/08 18:17:15 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 8 took 7 ms
> 16/06/08 18:17:16 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 10 took 8 ms
> 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 12 took 6 ms
> 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 13 took 7 ms
> ```
>
> after 23 hours:
>
> ```
> 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282938 took 18 ms
> 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282939 took 20 ms
> 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282940 took 20 ms
> 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282941 took 14 ms
> 16/06/09 17:23:44 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282942 took 17 ms
> 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282943 took 18 ms
> 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282944 took 14 ms
> 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282945 took 18 ms
> 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282946 took 14 ms
> 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast
> variable 282947 took 18 ms
> ```
>
> FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client mode.
>
> application environment follows:
>
> ```
> Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre
> Java Version 1.7.0_101 (Oracle Corporation)
> Scala Version version 2.10.5
>
> spark.app.id application_1463430559850_0056
> spark.cleaner.ttl 60
> spark.default.parallelism 96
> spark.driver.appUIAddress http://172.16.4.168:4041
> spark.driver.extraClassPath
> /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.driver.extraJavaOptions
> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled
> -XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p'
> spark.driver.extraLibraryPath
> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.driver.host 172.16.4.168
> spark.driver.port 42142
> spark.dynamicAllocation.enabled false
> spark.eventLog.compress true
> spark.eventLog.dir hdfs:///var/log/spark/apps
> spark.eventLog.enabled false
> spark.executor.cores 8
> spark.executor.extraClassPath
> /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70
> -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
> spark.executor.extraLibraryPath
> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.executor.id driver
> spark.executor.instances 24
> spark.executor.memory 6G
> spark.externalBlockStore.folderName spark-9ede6685-a7f9-4d65-bfb4-0920e8c5ba25
> spark.history.fs.cleaner.enabled true
> spark.history.fs.cleaner.maxAge 2d
> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
> spark.history.ui.port 18080
> spark.master yarn-client
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS
> ip-172-16-4-168.ec2.internal
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES
> http://ip-172-16-4-168.ec2.internal:20888/proxy/application_1463430559850_0056
> spark.scheduler.mode FAIR
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.shuffle.service.enabled false
> spark.streaming.backpressure.enabled true
> spark.streaming.concurrentJobs 2
> spark.streaming.kafka.maxRatePerPartition 312
> spark.streaming.ui.retainedBatches 1000
> spark.submit.deployMode client
> spark.ui.filters org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> spark.ui.retainedJobs 1000
> spark.ui.retainedStages 1000
> spark.yarn.executor.memoryOverhead 4096
> spark.yarn.historyServer.address ip-172-16-4-168.ec2.internal:18080
>
> SPARK_SUBMIT true
> SPARK_YARN_MODE true
> awt.toolkit sun.awt.X11.XToolkit
> file.encoding UTF-8
> file.encoding.pkg sun.io
> file.separator /
> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
> java.awt.printerjob sun.print.PSPrinterJob
> java.class.version 51.0
> java.endorsed.dirs
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/endorsed
> java.ext.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/ext:/usr/java/packages/lib/ext
> java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre
> java.io.tmpdir /tmp
> java.library.path
> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> java.runtime.name OpenJDK Runtime Environment
> java.runtime.version 1.7.0_101-mockbuild_2016_04_26_18_10-b00
> java.specification.name Java Platform API Specification
> java.specification.vendor Oracle Corporation
> java.specification.version 1.7
> java.vendor Oracle Corporation
> java.vendor.url http://java.oracle.com/
> java.vendor.url.bug http://bugreport.sun.com/bugreport/
> java.version 1.7.0_101
> java.vm.info mixed mode
> java.vm.name OpenJDK 64-Bit Server VM
> java.vm.specification.name Java Virtual Machine Specification
> java.vm.specification.vendor Oracle Corporation
> java.vm.specification.version 1.7
> java.vm.vendor Oracle Corporation
> java.vm.version 24.95-b01
> line.separator
> log4j.configuration file:///etc/spark/conf/log4j.properties
> os.arch amd64
> os.name Linux
> os.version 4.4.5-15.26.amzn1.x86_64
> path.separator :
> sun.arch.data.model 64
> sun.boot.class.path
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/classes
> sun.boot.library.path
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/amd64
> sun.cpu.endian little
> sun.cpu.isalist
> sun.io.unicode.encoding UnicodeLittle
> sun.java.launcher SUN_STANDARD
> sun.jnu.encoding UTF-8
> sun.management.compiler HotSpot 64-Bit Tiered Compilers
> sun.nio.ch.bugLevel
> sun.os.patch.level unknown
> user.country US
> user.dir /home/hadoop
> user.home /home/hadoop
> user.language en
> user.name hadoop
> user.timezone UTC
> ```
>
>
> --
> John Simon

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org