You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bartek (JIRA)" <ji...@apache.org> on 2015/06/18 13:51:00 UTC
[jira] [Created] (SPARK-8440) kafka spark steaming working example
Bartek created SPARK-8440:
-----------------------------
Summary: kafka spark steaming working example
Key: SPARK-8440
URL: https://issues.apache.org/jira/browse/SPARK-8440
Project: Spark
Issue Type: Question
Components: Streaming
Affects Versions: 1.4.0, 1.2.2
Reporter: Bartek
hi,
I'm trying to run simple kafka spark streaming example over spark-shell:
{code}
sc.stop
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext._
import kafka.serializer.DefaultDecoder
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
import org.apache.spark.storage.StorageLevel
val sparkConf = new SparkConf().setAppName("Summarizer").setMaster("local")
val ssc = new StreamingContext(sparkConf, Seconds(10))
val kafkaParams = Map[String, String]("zookeeper.connect" -> "127.0.0.1:2181", "group.id" -> "test")
val messages = KafkaUtils.createStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder](ssc, kafkaParams, Map("test" -> 1), StorageLevel.MEMORY_ONLY_SER).map(_._2)
messages.foreachRDD { pairRDD =>
println(s"DataListener.listen() [pairRDD = ${pairRDD}]")
println(s"DataListener.listen() [pairRDD.count = ${pairRDD.count()}]")
pairRDD.foreach(row => println(s"DataListener.listen() [row = ${row}]"))
}
ssc.start()
ssc.awaitTermination()
{code}
in spark output i'm able to find log {{println(s"DataListener.listen() [pairRDD = ${pairRDD}]")}} but can't find output of {{println(s"DataListener.listen() [pairRDD.count = ${pairRDD.count()}]")}} and {{println(s"DataListener.listen() [row = ${row}]")}}
it's my spark-shell full output:
{quote}
br@debian-jessie:~/spark-1.4.0-bin-hadoop1/bin$ ./spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/06/18 13:45:02 INFO SecurityManager: Changing view acls to: br
15/06/18 13:45:02 INFO SecurityManager: Changing modify acls to: br
15/06/18 13:45:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(br); users with modify permissions: Set(br)
15/06/18 13:45:02 INFO HttpServer: Starting HTTP Server
15/06/18 13:45:02 INFO Utils: Successfully started service 'HTTP class server' on port 38592.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/06/18 13:45:05 INFO SparkContext: Running Spark version 1.4.0
15/06/18 13:45:05 WARN SparkConf:
SPARK_CLASSPATH was detected (set to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
15/06/18 13:45:05 WARN SparkConf: Setting 'spark.executor.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:05 WARN SparkConf: Setting 'spark.driver.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:05 INFO SecurityManager: Changing view acls to: br
15/06/18 13:45:05 INFO SecurityManager: Changing modify acls to: br
15/06/18 13:45:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(br); users with modify permissions: Set(br)
15/06/18 13:45:05 INFO Slf4jLogger: Slf4jLogger started
15/06/18 13:45:05 INFO Remoting: Starting remoting
15/06/18 13:45:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.168.130:50650]
15/06/18 13:45:05 INFO Utils: Successfully started service 'sparkDriver' on port 50650.
15/06/18 13:45:05 INFO SparkEnv: Registering MapOutputTracker
15/06/18 13:45:05 INFO SparkEnv: Registering BlockManagerMaster
15/06/18 13:45:06 INFO DiskBlockManager: Created local directory at /tmp/spark-7058feff-0c15-4af6-aeaf-47a21ec594ec/blockmgr-7c0070f8-0534-4156-963f-af56c5326a70
15/06/18 13:45:06 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
15/06/18 13:45:06 INFO HttpFileServer: HTTP File server directory is /tmp/spark-7058feff-0c15-4af6-aeaf-47a21ec594ec/httpd-7d913234-bf93-434f-a351-09abc2a45dc0
15/06/18 13:45:06 INFO HttpServer: Starting HTTP Server
15/06/18 13:45:06 INFO Utils: Successfully started service 'HTTP file server' on port 35480.
15/06/18 13:45:06 INFO SparkEnv: Registering OutputCommitCoordinator
15/06/18 13:45:06 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/06/18 13:45:06 INFO SparkUI: Started SparkUI at http://192.168.168.130:4040
15/06/18 13:45:06 INFO Executor: Starting executor ID driver on host localhost
15/06/18 13:45:06 INFO Executor: Using REPL class URI: http://192.168.168.130:38592
15/06/18 13:45:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49412.
15/06/18 13:45:06 INFO NettyBlockTransferService: Server created on 49412
15/06/18 13:45:06 INFO BlockManagerMaster: Trying to register BlockManager
15/06/18 13:45:06 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49412 with 265.1 MB RAM, BlockManagerId(driver, localhost, 49412)
15/06/18 13:45:06 INFO BlockManagerMaster: Registered BlockManager
15/06/18 13:45:06 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/06/18 13:45:06 INFO HiveContext: Initializing execution hive, version 0.13.1
15/06/18 13:45:07 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/06/18 13:45:07 INFO ObjectStore: ObjectStore, initialize called
15/06/18 13:45:07 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/06/18 13:45:07 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/06/18 13:45:07 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/06/18 13:45:07 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/06/18 13:45:08 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/06/18 13:45:08 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/06/18 13:45:09 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:09 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:10 INFO ObjectStore: Initialized ObjectStore
15/06/18 13:45:10 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/06/18 13:45:10 INFO HiveMetaStore: Added admin role in metastore
15/06/18 13:45:10 INFO HiveMetaStore: Added public role in metastore
15/06/18 13:45:10 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/06/18 13:45:10 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/06/18 13:45:10 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> 15/06/18 13:45:15 INFO SparkContext: Invoking stop() from shutdown hook
15/06/18 13:45:15 INFO SparkUI: Stopped Spark web UI at http://192.168.168.130:4040
15/06/18 13:45:15 INFO DAGScheduler: Stopping DAGScheduler
15/06/18 13:45:15 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/06/18 13:45:15 INFO Utils: path = /tmp/spark-7058feff-0c15-4af6-aeaf-47a21ec594ec/blockmgr-7c0070f8-0534-4156-963f-af56c5326a70, already present as root for deletion.
15/06/18 13:45:15 INFO MemoryStore: MemoryStore cleared
15/06/18 13:45:15 INFO BlockManager: BlockManager stopped
15/06/18 13:45:15 INFO BlockManagerMaster: BlockManagerMaster stopped
15/06/18 13:45:15 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/06/18 13:45:15 INFO SparkContext: Successfully stopped SparkContext
15/06/18 13:45:15 INFO Utils: Shutdown hook called
15/06/18 13:45:15 INFO Utils: Deleting directory /tmp/spark-7058feff-0c15-4af6-aeaf-47a21ec594ec
15/06/18 13:45:15 INFO Utils: Deleting directory /tmp/spark-3e878219-2a0d-4843-8dbd-f1205ce43bd2
15/06/18 13:45:15 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/06/18 13:45:15 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/06/18 13:45:15 INFO Utils: Deleting directory /tmp/spark-ed5c9445-63cb-46e5-be13-18cbb4352f79
br@debian-jessie:~/spark-1.4.0-bin-hadoop1/bin$ export SPARK_CLASSPATH="$(echo /home/br/spark-1.4.0-bin-hadoop1/lib/addons/*.jar |sed 's/ /:/g'):"
br@debian-jessie:~/spark-1.4.0-bin-hadoop1/bin$ ./spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/06/18 13:45:20 INFO SecurityManager: Changing view acls to: br
15/06/18 13:45:20 INFO SecurityManager: Changing modify acls to: br
15/06/18 13:45:20 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(br); users with modify permissions: Set(br)
15/06/18 13:45:20 INFO HttpServer: Starting HTTP Server
15/06/18 13:45:20 INFO Utils: Successfully started service 'HTTP class server' on port 53798.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/06/18 13:45:22 INFO SparkContext: Running Spark version 1.4.0
15/06/18 13:45:22 WARN SparkConf:
SPARK_CLASSPATH was detected (set to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
15/06/18 13:45:22 WARN SparkConf: Setting 'spark.executor.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:22 WARN SparkConf: Setting 'spark.driver.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:22 INFO SecurityManager: Changing view acls to: br
15/06/18 13:45:22 INFO SecurityManager: Changing modify acls to: br
15/06/18 13:45:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(br); users with modify permissions: Set(br)
15/06/18 13:45:22 INFO Slf4jLogger: Slf4jLogger started
15/06/18 13:45:22 INFO Remoting: Starting remoting
15/06/18 13:45:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.168.130:41375]
15/06/18 13:45:23 INFO Utils: Successfully started service 'sparkDriver' on port 41375.
15/06/18 13:45:23 INFO SparkEnv: Registering MapOutputTracker
15/06/18 13:45:23 INFO SparkEnv: Registering BlockManagerMaster
15/06/18 13:45:23 INFO DiskBlockManager: Created local directory at /tmp/spark-97102888-de72-4084-b2b5-494b1c630545/blockmgr-bccea169-b161-43b3-bb60-85027a9b0f70
15/06/18 13:45:23 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
15/06/18 13:45:23 INFO HttpFileServer: HTTP File server directory is /tmp/spark-97102888-de72-4084-b2b5-494b1c630545/httpd-fa627bcc-f130-4881-a8e9-9f5b85c8aca6
15/06/18 13:45:23 INFO HttpServer: Starting HTTP Server
15/06/18 13:45:23 INFO Utils: Successfully started service 'HTTP file server' on port 48632.
15/06/18 13:45:23 INFO SparkEnv: Registering OutputCommitCoordinator
15/06/18 13:45:23 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/06/18 13:45:23 INFO SparkUI: Started SparkUI at http://192.168.168.130:4040
15/06/18 13:45:23 INFO Executor: Starting executor ID driver on host localhost
15/06/18 13:45:23 INFO Executor: Using REPL class URI: http://192.168.168.130:53798
15/06/18 13:45:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59971.
15/06/18 13:45:23 INFO NettyBlockTransferService: Server created on 59971
15/06/18 13:45:23 INFO BlockManagerMaster: Trying to register BlockManager
15/06/18 13:45:23 INFO BlockManagerMasterEndpoint: Registering block manager localhost:59971 with 265.1 MB RAM, BlockManagerId(driver, localhost, 59971)
15/06/18 13:45:23 INFO BlockManagerMaster: Registered BlockManager
15/06/18 13:45:23 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/06/18 13:45:24 INFO HiveContext: Initializing execution hive, version 0.13.1
15/06/18 13:45:24 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/06/18 13:45:24 INFO ObjectStore: ObjectStore, initialize called
15/06/18 13:45:24 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/06/18 13:45:24 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/06/18 13:45:24 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/06/18 13:45:25 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/06/18 13:45:25 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/06/18 13:45:26 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/06/18 13:45:26 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:26 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:27 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:27 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/06/18 13:45:27 INFO ObjectStore: Initialized ObjectStore
15/06/18 13:45:27 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/06/18 13:45:27 INFO HiveMetaStore: Added admin role in metastore
15/06/18 13:45:27 INFO HiveMetaStore: Added public role in metastore
15/06/18 13:45:27 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/06/18 13:45:27 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/06/18 13:45:27 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> sc.stop
15/06/18 13:45:30 INFO SparkUI: Stopped Spark web UI at http://192.168.168.130:4040
15/06/18 13:45:30 INFO DAGScheduler: Stopping DAGScheduler
15/06/18 13:45:30 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/06/18 13:45:30 INFO Utils: path = /tmp/spark-97102888-de72-4084-b2b5-494b1c630545/blockmgr-bccea169-b161-43b3-bb60-85027a9b0f70, already present as root for deletion.
15/06/18 13:45:30 INFO MemoryStore: MemoryStore cleared
15/06/18 13:45:30 INFO BlockManager: BlockManager stopped
15/06/18 13:45:30 INFO BlockManagerMaster: BlockManagerMaster stopped
15/06/18 13:45:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/06/18 13:45:30 INFO SparkContext: Successfully stopped SparkContext
scala> imp15/06/18 13:45:30 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/06/18 13:45:30 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
ort org.apache.spark.SparkConf
15/06/18 13:45:30 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
import org.apache.spark.SparkConf
scala> import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext._
scala> import kafka.serializer.DefaultDecoder
import kafka.serializer.DefaultDecoder
scala> import org.apache.spark.streaming._
import org.apache.spark.streaming._
scala> import org.apache.spark.streaming.kafka._
import org.apache.spark.streaming.kafka._
scala> import org.apache.spark.storage.StorageLevel
import org.apache.spark.storage.StorageLevel
scala> val sparkConf = new SparkConf().setAppName("Summarizer").setMaster("local")
sparkConf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@4537c9f8
scala> val ssc = new StreamingContext(sparkConf, Seconds(10))
15/06/18 13:45:32 INFO SparkContext: Running Spark version 1.4.0
15/06/18 13:45:32 WARN SparkConf:
SPARK_CLASSPATH was detected (set to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
15/06/18 13:45:32 WARN SparkConf: Setting 'spark.executor.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:32 WARN SparkConf: Setting 'spark.driver.extraClassPath' to '/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:' as a work-around.
15/06/18 13:45:32 INFO SecurityManager: Changing view acls to: br
15/06/18 13:45:32 INFO SecurityManager: Changing modify acls to: br
15/06/18 13:45:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(br); users with modify permissions: Set(br)
15/06/18 13:45:32 INFO Slf4jLogger: Slf4jLogger started
15/06/18 13:45:32 INFO Remoting: Starting remoting
15/06/18 13:45:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:52075]
15/06/18 13:45:32 INFO Utils: Successfully started service 'sparkDriver' on port 52075.
15/06/18 13:45:32 INFO SparkEnv: Registering MapOutputTracker
15/06/18 13:45:32 INFO SparkEnv: Registering BlockManagerMaster
15/06/18 13:45:32 INFO DiskBlockManager: Created local directory at /tmp/spark-97102888-de72-4084-b2b5-494b1c630545/blockmgr-05b26ff9-10d2-4fa7-a1d7-e7a76dabdc34
15/06/18 13:45:32 INFO MemoryStore: MemoryStore started with capacity 247.3 MB
15/06/18 13:45:32 INFO HttpFileServer: HTTP File server directory is /tmp/spark-97102888-de72-4084-b2b5-494b1c630545/httpd-a86723a3-1932-4e4b-a641-e6fb484e2594
15/06/18 13:45:32 INFO HttpServer: Starting HTTP Server
15/06/18 13:45:32 INFO Utils: Successfully started service 'HTTP file server' on port 44790.
15/06/18 13:45:32 INFO SparkEnv: Registering OutputCommitCoordinator
15/06/18 13:45:32 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/06/18 13:45:32 INFO SparkUI: Started SparkUI at http://localhost:4040
15/06/18 13:45:32 INFO Executor: Starting executor ID driver on host localhost
15/06/18 13:45:32 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49443.
15/06/18 13:45:32 INFO NettyBlockTransferService: Server created on 49443
15/06/18 13:45:32 INFO BlockManagerMaster: Trying to register BlockManager
15/06/18 13:45:32 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49443 with 247.3 MB RAM, BlockManagerId(driver, localhost, 49443)
15/06/18 13:45:32 INFO BlockManagerMaster: Registered BlockManager
15/06/18 13:45:32 WARN StreamingContext: spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.
ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@4fe03c06
scala> val kafkaParams = Map[String, String]("zookeeper.connect" -> "127.0.0.1:2181", "group.id" -> "test")
kafkaParams: scala.collection.immutable.Map[String,String] = Map(zookeeper.connect -> 127.0.0.1:2181, group.id -> test)
scala> val messages = KafkaUtils.createStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder](ssc, kafkaParams, Map("test" -> 1), StorageLevel.MEMORY_ONLY_SER).map(_._2)
messages: org.apache.spark.streaming.dstream.DStream[Array[Byte]] = org.apache.spark.streaming.dstream.MappedDStream@10b40e87
scala>
scala> messages.foreachRDD { pairRDD =>
| println(s"DataListener.listen() [pairRDD = ${pairRDD}]")
| println(s"DataListener.listen() [pairRDD.count = ${pairRDD.count()}]")
| pairRDD.foreach(row => println(s"DataListener.listen() [row = ${row}]"))
| }
scala>
scala> ssc.start()
15/06/18 13:45:34 INFO ReceiverTracker: ReceiverTracker started
15/06/18 13:45:34 INFO ForEachDStream: metadataCleanupDelay = -1
15/06/18 13:45:34 INFO MappedDStream: metadataCleanupDelay = -1
15/06/18 13:45:34 INFO KafkaInputDStream: metadataCleanupDelay = -1
15/06/18 13:45:34 INFO KafkaInputDStream: Slide time = 10000 ms
15/06/18 13:45:34 INFO KafkaInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/06/18 13:45:34 INFO KafkaInputDStream: Checkpoint interval = null
15/06/18 13:45:34 INFO KafkaInputDStream: Remember duration = 10000 ms
15/06/18 13:45:34 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@2e00032d
15/06/18 13:45:34 INFO MappedDStream: Slide time = 10000 ms
15/06/18 13:45:34 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/06/18 13:45:34 INFO MappedDStream: Checkpoint interval = null
15/06/18 13:45:34 INFO MappedDStream: Remember duration = 10000 ms
15/06/18 13:45:34 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@10b40e87
15/06/18 13:45:34 INFO ForEachDStream: Slide time = 10000 ms
15/06/18 13:45:34 INFO ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/06/18 13:45:34 INFO ForEachDStream: Checkpoint interval = null
15/06/18 13:45:34 INFO ForEachDStream: Remember duration = 10000 ms
15/06/18 13:45:34 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@3934574f
15/06/18 13:45:34 INFO ReceiverTracker: Starting 1 receivers
15/06/18 13:45:34 INFO SparkContext: Starting job: start at <console>:36
15/06/18 13:45:34 INFO DAGScheduler: Got job 0 (start at <console>:36) with 1 output partitions (allowLocal=false)
15/06/18 13:45:34 INFO DAGScheduler: Final stage: ResultStage 0(start at <console>:36)
15/06/18 13:45:34 INFO DAGScheduler: Parents of final stage: List()
15/06/18 13:45:34 INFO DAGScheduler: Missing parents: List()
15/06/18 13:45:34 INFO DAGScheduler: Submitting ResultStage 0 (ParallelCollectionRDD[0] at start at <console>:36), which has no missing parents
15/06/18 13:45:34 INFO RecurringTimer: Started timer for JobGenerator at time 1434627940000
15/06/18 13:45:34 INFO JobGenerator: Started JobGenerator at 1434627940000 ms
15/06/18 13:45:34 INFO JobScheduler: Started JobScheduler
15/06/18 13:45:34 INFO StreamingContext: StreamingContext started
scala> ssc.awaitTermination()
15/06/18 13:45:34 INFO MemoryStore: ensureFreeSpace(7952) called with curMem=0, maxMem=259333816
15/06/18 13:45:34 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 7.8 KB, free 247.3 MB)
15/06/18 13:45:34 INFO MemoryStore: ensureFreeSpace(4451) called with curMem=7952, maxMem=259333816
15/06/18 13:45:34 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.3 KB, free 247.3 MB)
15/06/18 13:45:34 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:49443 (size: 4.3 KB, free: 247.3 MB)
15/06/18 13:45:34 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:874
15/06/18 13:45:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (ParallelCollectionRDD[0] at start at <console>:36)
15/06/18 13:45:34 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/06/18 13:45:34 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 2313 bytes)
15/06/18 13:45:34 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/06/18 13:45:35 INFO RecurringTimer: Started timer for BlockGenerator at time 1434627935200
15/06/18 13:45:35 INFO BlockGenerator: Started BlockGenerator
15/06/18 13:45:35 INFO ReceiverSupervisorImpl: Starting receiver
15/06/18 13:45:35 INFO KafkaReceiver: Starting Kafka Consumer Stream with group: test
15/06/18 13:45:35 INFO KafkaReceiver: Connecting to Zookeeper: 127.0.0.1:2181
15/06/18 13:45:35 INFO BlockGenerator: Started block pushing thread
15/06/18 13:45:35 INFO VerifiableProperties: Verifying properties
15/06/18 13:45:35 INFO VerifiableProperties: Property group.id is overridden to test
15/06/18 13:45:35 INFO VerifiableProperties: Property zookeeper.connect is overridden to 127.0.0.1:2181
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Connecting to zookeeper instance at 127.0.0.1:2181
15/06/18 13:45:35 INFO ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
15/06/18 13:45:35 INFO ZooKeeper: Client environment:host.name=debian-jessie
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.version=1.8.0_45
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.class.path=/home/br/spark-1.4.0-bin-hadoop1/lib/addons/kafka_2.10-0.8.1.1.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/metrics-core-2.2.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/spark-streaming-kafka_2.10-1.4.0.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/addons/zkclient-0.5.jar:/home/br/spark-1.4.0-bin-hadoop1/conf/:/home/br/spark-1.4.0-bin-hadoop1/lib/spark-assembly-1.4.0-hadoop1.0.4.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/datanucleus-core-3.2.10.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/datanucleus-api-jdo-3.2.6.jar:/home/br/spark-1.4.0-bin-hadoop1/lib/datanucleus-rdbms-3.2.9.jar
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
15/06/18 13:45:35 INFO ZooKeeper: Client environment:java.compiler=<NA>
15/06/18 13:45:35 INFO ZooKeeper: Client environment:os.name=Linux
15/06/18 13:45:35 INFO ZooKeeper: Client environment:os.arch=amd64
15/06/18 13:45:35 INFO ZooKeeper: Client environment:os.version=3.16.0-4-amd64
15/06/18 13:45:35 INFO ZooKeeper: Client environment:user.name=br
15/06/18 13:45:35 INFO ZooKeeper: Client environment:user.home=/home/br
15/06/18 13:45:35 INFO ZooKeeper: Client environment:user.dir=/home/br/spark-1.4.0-bin-hadoop1/bin
15/06/18 13:45:35 INFO ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@277025ec
15/06/18 13:45:35 INFO ZkEventThread: Starting ZkClient event thread.
15/06/18 13:45:35 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
15/06/18 13:45:35 INFO ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
15/06/18 13:45:35 WARN ClientCnxnSocket: Connected to an old server; r-o mode will be unavailable
15/06/18 13:45:35 INFO ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14e067c68f10001, negotiated timeout = 6000
15/06/18 13:45:35 INFO ZkClient: zookeeper state changed (SyncConnected)
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], starting auto committer every 60000 ms
15/06/18 13:45:35 INFO KafkaReceiver: Connected to 127.0.0.1:2181
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], begin registering consumer test_debian-jessie-1434627935036-fa2d3110 in ZK
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], end registering consumer test_debian-jessie-1434627935036-fa2d3110 in ZK
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], starting watcher executor thread for consumer test_debian-jessie-1434627935036-fa2d3110
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], begin rebalancing consumer test_debian-jessie-1434627935036-fa2d3110 try #0
15/06/18 13:45:35 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1434627935098] Stopping leader finder thread
15/06/18 13:45:35 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1434627935098] Stopping all fetchers
15/06/18 13:45:35 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1434627935098] All connections stopped
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Cleared all relevant queues for this fetcher
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Cleared the data chunks in all the consumer message iterators
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Committing all offsets after clearing the fetcher queues
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Releasing partition ownership
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Consumer test_debian-jessie-1434627935036-fa2d3110 rebalancing the following partitions: ArrayBuffer(0) for topic test with consumers: List(test_debian-jessie-1434627935036-fa2d3110-0)
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], test_debian-jessie-1434627935036-fa2d3110-0 attempting to claim partition 0
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], test_debian-jessie-1434627935036-fa2d3110-0 successfully owned partition 0 for topic test
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Updating the cache
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], Consumer test_debian-jessie-1434627935036-fa2d3110 selected partitions : test:0: fetched offset = 104349: consumed offset = 104349
15/06/18 13:45:35 INFO ZookeeperConsumerConnector: [test_debian-jessie-1434627935036-fa2d3110], end rebalancing consumer test_debian-jessie-1434627935036-fa2d3110 try #0
15/06/18 13:45:35 INFO ReceiverSupervisorImpl: Called receiver onStart
15/06/18 13:45:35 INFO ReceiverTracker: Registered receiver for stream 0 from localhost:52075
15/06/18 13:45:35 INFO KafkaReceiver: Starting MessageHandler.
15/06/18 13:45:35 INFO ConsumerFetcherManager$LeaderFinderThread: [test_debian-jessie-1434627935036-fa2d3110-leader-finder-thread], Starting
15/06/18 13:45:35 INFO VerifiableProperties: Verifying properties
15/06/18 13:45:35 INFO VerifiableProperties: Property client.id is overridden to test
15/06/18 13:45:35 INFO VerifiableProperties: Property metadata.broker.list is overridden to debian-jessie:9092
15/06/18 13:45:35 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
15/06/18 13:45:35 INFO ClientUtils$: Fetching metadata from broker id:0,host:debian-jessie,port:9092 with correlation id 0 for 1 topic(s) Set(test)
15/06/18 13:45:35 INFO SyncProducer: Connected to debian-jessie:9092 for producing
15/06/18 13:45:35 INFO SyncProducer: Disconnecting from debian-jessie:9092
15/06/18 13:45:35 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1434627935098] Added fetcher for partitions ArrayBuffer([[test,0], initOffset 104349 to broker id:0,host:debian-jessie,port:9092] )
15/06/18 13:45:35 INFO ConsumerFetcherThread: [ConsumerFetcherThread-test_debian-jessie-1434627935036-fa2d3110-0-0], Starting
15/06/18 13:45:40 INFO JobScheduler: Added jobs for time 1434627940000 ms
15/06/18 13:45:40 INFO JobScheduler: Starting job streaming job 1434627940000 ms.0 from job set of time 1434627940000 ms
DataListener.listen() [pairRDD = MapPartitionsRDD[2] at map at <console>:37]
15/06/18 13:45:40 INFO SparkContext: Starting job: foreachRDD at <console>:40
15/06/18 13:45:40 INFO DAGScheduler: Job 1 finished: foreachRDD at <console>:40, took 0.000688 s
DataListener.listen() [pairRDD.count = 0]
15/06/18 13:45:40 INFO SparkContext: Starting job: foreachRDD at <console>:40
15/06/18 13:45:40 INFO DAGScheduler: Job 2 finished: foreachRDD at <console>:40, took 0.000184 s
15/06/18 13:45:40 INFO JobScheduler: Finished job streaming job 1434627940000 ms.0 from job set of time 1434627940000 ms
15/06/18 13:45:40 INFO JobScheduler: Total delay: 0.051 s for time 1434627940000 ms (execution: 0.015 s)
15/06/18 13:45:40 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()
15/06/18 13:45:40 INFO InputInfoTracker: remove old batch metadata:
15/06/18 13:45:44 INFO MemoryStore: ensureFreeSpace(2426) called with curMem=12403, maxMem=259333816
15/06/18 13:45:44 INFO MemoryStore: Block input-0-1434627943800 stored as bytes in memory (estimated size 2.4 KB, free 247.3 MB)
15/06/18 13:45:44 INFO BlockManagerInfo: Added input-0-1434627943800 in memory on localhost:49443 (size: 2.4 KB, free: 247.3 MB)
15/06/18 13:45:44 INFO BlockGenerator: Pushed block input-0-1434627943800
15/06/18 13:45:44 INFO MemoryStore: ensureFreeSpace(526) called with curMem=14829, maxMem=259333816
15/06/18 13:45:44 INFO MemoryStore: Block input-0-1434627944000 stored as bytes in memory (estimated size 526.0 B, free 247.3 MB)
15/06/18 13:45:44 INFO BlockManagerInfo: Added input-0-1434627944000 in memory on localhost:49443 (size: 526.0 B, free: 247.3 MB)
15/06/18 13:45:44 INFO BlockGenerator: Pushed block input-0-1434627944000
15/06/18 13:45:44 INFO MemoryStore: ensureFreeSpace(10568) called with curMem=15355, maxMem=259333816
15/06/18 13:45:44 INFO MemoryStore: Block input-0-1434627944400 stored as bytes in memory (estimated size 10.3 KB, free 247.3 MB)
15/06/18 13:45:44 INFO BlockManagerInfo: Added input-0-1434627944400 in memory on localhost:49443 (size: 10.3 KB, free: 247.3 MB)
15/06/18 13:45:44 INFO BlockGenerator: Pushed block input-0-1434627944400
15/06/18 13:45:44 INFO MemoryStore: ensureFreeSpace(3899) called with curMem=25923, maxMem=259333816
15/06/18 13:45:44 INFO MemoryStore: Block input-0-1434627944600 stored as bytes in memory (estimated size 3.8 KB, free 247.3 MB)
15/06/18 13:45:44 INFO BlockManagerInfo: Added input-0-1434627944600 in memory on localhost:49443 (size: 3.8 KB, free: 247.3 MB)
15/06/18 13:45:44 INFO BlockGenerator: Pushed block input-0-1434627944600
15/06/18 13:45:45 INFO MemoryStore: ensureFreeSpace(3249) called with curMem=29822, maxMem=259333816
15/06/18 13:45:45 INFO MemoryStore: Block input-0-1434627944800 stored as bytes in memory (estimated size 3.2 KB, free 247.3 MB)
15/06/18 13:45:45 INFO BlockManagerInfo: Added input-0-1434627944800 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:45:45 INFO BlockGenerator: Pushed block input-0-1434627944800
15/06/18 13:45:45 INFO MemoryStore: ensureFreeSpace(3199) called with curMem=33071, maxMem=259333816
15/06/18 13:45:45 INFO MemoryStore: Block input-0-1434627945000 stored as bytes in memory (estimated size 3.1 KB, free 247.3 MB)
15/06/18 13:45:45 INFO BlockManagerInfo: Added input-0-1434627945000 in memory on localhost:49443 (size: 3.1 KB, free: 247.3 MB)
15/06/18 13:45:45 INFO BlockGenerator: Pushed block input-0-1434627945000
15/06/18 13:45:45 INFO MemoryStore: ensureFreeSpace(3249) called with curMem=36270, maxMem=259333816
15/06/18 13:45:45 INFO MemoryStore: Block input-0-1434627945200 stored as bytes in memory (estimated size 3.2 KB, free 247.3 MB)
15/06/18 13:45:45 INFO BlockManagerInfo: Added input-0-1434627945200 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:45:45 INFO BlockGenerator: Pushed block input-0-1434627945200
15/06/18 13:45:45 INFO MemoryStore: ensureFreeSpace(3249) called with curMem=39519, maxMem=259333816
15/06/18 13:45:45 INFO MemoryStore: Block input-0-1434627945400 stored as bytes in memory (estimated size 3.2 KB, free 247.3 MB)
15/06/18 13:45:45 INFO BlockManagerInfo: Added input-0-1434627945400 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:45:45 INFO BlockGenerator: Pushed block input-0-1434627945400
15/06/18 13:45:45 INFO MemoryStore: ensureFreeSpace(3249) called with curMem=42768, maxMem=259333816
15/06/18 13:45:45 INFO MemoryStore: Block input-0-1434627945600 stored as bytes in memory (estimated size 3.2 KB, free 247.3 MB)
15/06/18 13:45:45 INFO BlockManagerInfo: Added input-0-1434627945600 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:45:45 INFO BlockGenerator: Pushed block input-0-1434627945600
15/06/18 13:45:46 INFO MemoryStore: ensureFreeSpace(3299) called with curMem=46017, maxMem=259333816
15/06/18 13:45:46 INFO MemoryStore: Block input-0-1434627945800 stored as bytes in memory (estimated size 3.2 KB, free 247.3 MB)
15/06/18 13:45:46 INFO BlockManagerInfo: Added input-0-1434627945800 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:45:46 INFO BlockGenerator: Pushed block input-0-1434627945800
15/06/18 13:45:46 INFO MemoryStore: ensureFreeSpace(2076) called with curMem=49316, maxMem=259333816
15/06/18 13:45:46 INFO MemoryStore: Block input-0-1434627946000 stored as bytes in memory (estimated size 2.0 KB, free 247.3 MB)
15/06/18 13:45:46 INFO BlockManagerInfo: Added input-0-1434627946000 in memory on localhost:49443 (size: 2.0 KB, free: 247.3 MB)
15/06/18 13:45:46 INFO BlockGenerator: Pushed block input-0-1434627946000
15/06/18 13:45:50 INFO JobScheduler: Starting job streaming job 1434627950000 ms.0 from job set of time 1434627950000 ms
DataListener.listen() [pairRDD = MapPartitionsRDD[4] at map at <console>:37]
15/06/18 13:45:50 INFO SparkContext: Starting job: foreachRDD at <console>:40
15/06/18 13:45:50 INFO JobScheduler: Added jobs for time 1434627950000 ms
15/06/18 13:45:50 INFO DAGScheduler: Got job 3 (foreachRDD at <console>:40) with 11 output partitions (allowLocal=false)
15/06/18 13:45:50 INFO DAGScheduler: Final stage: ResultStage 1(foreachRDD at <console>:40)
15/06/18 13:45:50 INFO DAGScheduler: Parents of final stage: List()
15/06/18 13:45:50 INFO DAGScheduler: Missing parents: List()
15/06/18 13:45:50 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[4] at map at <console>:37), which has no missing parents
15/06/18 13:45:50 INFO MemoryStore: ensureFreeSpace(1608) called with curMem=51392, maxMem=259333816
15/06/18 13:45:50 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1608.0 B, free 247.3 MB)
15/06/18 13:45:50 INFO MemoryStore: ensureFreeSpace(1046) called with curMem=53000, maxMem=259333816
15/06/18 13:45:50 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1046.0 B, free 247.3 MB)
15/06/18 13:45:50 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:49443 (size: 1046.0 B, free: 247.3 MB)
15/06/18 13:45:50 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
15/06/18 13:45:50 INFO DAGScheduler: Submitting 11 missing tasks from ResultStage 1 (MapPartitionsRDD[4] at map at <console>:37)
15/06/18 13:45:50 INFO TaskSchedulerImpl: Adding task set 1.0 with 11 tasks
15/06/18 13:46:00 INFO JobScheduler: Added jobs for time 1434627960000 ms
15/06/18 13:46:10 INFO JobScheduler: Added jobs for time 1434627970000 ms
15/06/18 13:46:20 INFO JobScheduler: Added jobs for time 1434627980000 ms
15/06/18 13:46:30 INFO JobScheduler: Added jobs for time 1434627990000 ms
15/06/18 13:46:40 INFO JobScheduler: Added jobs for time 1434628000000 ms
15/06/18 13:46:50 INFO JobScheduler: Added jobs for time 1434628010000 ms
15/06/18 13:47:00 INFO JobScheduler: Added jobs for time 1434628020000 ms
15/06/18 13:47:10 INFO JobScheduler: Added jobs for time 1434628030000 ms
15/06/18 13:47:20 INFO JobScheduler: Added jobs for time 1434628040000 ms
15/06/18 13:47:30 INFO JobScheduler: Added jobs for time 1434628050000 ms
15/06/18 13:47:40 INFO JobScheduler: Added jobs for time 1434628060000 ms
15/06/18 13:47:50 INFO JobScheduler: Added jobs for time 1434628070000 ms
15/06/18 13:48:00 INFO JobScheduler: Added jobs for time 1434628080000 ms
15/06/18 13:48:10 INFO JobScheduler: Added jobs for time 1434628090000 ms
15/06/18 13:48:20 INFO JobScheduler: Added jobs for time 1434628100000 ms
15/06/18 13:48:30 INFO JobScheduler: Added jobs for time 1434628110000 ms
15/06/18 13:48:40 INFO JobScheduler: Added jobs for time 1434628120000 ms
15/06/18 13:48:50 INFO JobScheduler: Added jobs for time 1434628130000 ms
15/06/18 13:49:00 INFO JobScheduler: Added jobs for time 1434628140000 ms
15/06/18 13:49:05 INFO MemoryStore: ensureFreeSpace(2899) called with curMem=54046, maxMem=259333816
15/06/18 13:49:05 INFO MemoryStore: Block input-0-1434628145400 stored as bytes in memory (estimated size 2.8 KB, free 247.3 MB)
15/06/18 13:49:05 INFO BlockManagerInfo: Added input-0-1434628145400 in memory on localhost:49443 (size: 2.8 KB, free: 247.3 MB)
15/06/18 13:49:05 INFO BlockGenerator: Pushed block input-0-1434628145400
15/06/18 13:49:05 INFO MemoryStore: ensureFreeSpace(3099) called with curMem=56945, maxMem=259333816
15/06/18 13:49:05 INFO MemoryStore: Block input-0-1434628145600 stored as bytes in memory (estimated size 3.0 KB, free 247.3 MB)
15/06/18 13:49:05 INFO BlockManagerInfo: Added input-0-1434628145600 in memory on localhost:49443 (size: 3.0 KB, free: 247.3 MB)
15/06/18 13:49:05 INFO BlockGenerator: Pushed block input-0-1434628145600
15/06/18 13:49:06 INFO MemoryStore: ensureFreeSpace(3449) called with curMem=60044, maxMem=259333816
15/06/18 13:49:06 INFO MemoryStore: Block input-0-1434628145800 stored as bytes in memory (estimated size 3.4 KB, free 247.3 MB)
15/06/18 13:49:06 INFO BlockManagerInfo: Added input-0-1434628145800 in memory on localhost:49443 (size: 3.4 KB, free: 247.3 MB)
15/06/18 13:49:06 INFO BlockGenerator: Pushed block input-0-1434628145800
15/06/18 13:49:06 INFO MemoryStore: ensureFreeSpace(3199) called with curMem=63493, maxMem=259333816
15/06/18 13:49:06 INFO MemoryStore: Block input-0-1434628146000 stored as bytes in memory (estimated size 3.1 KB, free 247.3 MB)
15/06/18 13:49:06 INFO BlockManagerInfo: Added input-0-1434628146000 in memory on localhost:49443 (size: 3.1 KB, free: 247.3 MB)
15/06/18 13:49:06 INFO BlockGenerator: Pushed block input-0-1434628146000
15/06/18 13:49:06 INFO MemoryStore: ensureFreeSpace(3049) called with curMem=66692, maxMem=259333816
15/06/18 13:49:06 INFO MemoryStore: Block input-0-1434628146200 stored as bytes in memory (estimated size 3.0 KB, free 247.3 MB)
15/06/18 13:49:06 INFO BlockManagerInfo: Added input-0-1434628146200 in memory on localhost:49443 (size: 3.0 KB, free: 247.3 MB)
15/06/18 13:49:06 INFO BlockGenerator: Pushed block input-0-1434628146200
15/06/18 13:49:06 INFO MemoryStore: ensureFreeSpace(3149) called with curMem=69741, maxMem=259333816
15/06/18 13:49:06 INFO MemoryStore: Block input-0-1434628146400 stored as bytes in memory (estimated size 3.1 KB, free 247.3 MB)
15/06/18 13:49:06 INFO BlockManagerInfo: Added input-0-1434628146400 in memory on localhost:49443 (size: 3.1 KB, free: 247.3 MB)
15/06/18 13:49:06 INFO BlockGenerator: Pushed block input-0-1434628146400
15/06/18 13:49:06 INFO MemoryStore: ensureFreeSpace(3049) called with curMem=72890, maxMem=259333816
15/06/18 13:49:06 INFO MemoryStore: Block input-0-1434628146600 stored as bytes in memory (estimated size 3.0 KB, free 247.2 MB)
15/06/18 13:49:06 INFO BlockManagerInfo: Added input-0-1434628146600 in memory on localhost:49443 (size: 3.0 KB, free: 247.3 MB)
15/06/18 13:49:06 INFO BlockGenerator: Pushed block input-0-1434628146600
15/06/18 13:49:07 INFO MemoryStore: ensureFreeSpace(3249) called with curMem=75939, maxMem=259333816
15/06/18 13:49:07 INFO MemoryStore: Block input-0-1434628146800 stored as bytes in memory (estimated size 3.2 KB, free 247.2 MB)
15/06/18 13:49:07 INFO BlockManagerInfo: Added input-0-1434628146800 in memory on localhost:49443 (size: 3.2 KB, free: 247.3 MB)
15/06/18 13:49:07 INFO BlockGenerator: Pushed block input-0-1434628146800
15/06/18 13:49:07 INFO MemoryStore: ensureFreeSpace(2476) called with curMem=79188, maxMem=259333816
15/06/18 13:49:07 INFO MemoryStore: Block input-0-1434628147000 stored as bytes in memory (estimated size 2.4 KB, free 247.2 MB)
15/06/18 13:49:07 INFO BlockManagerInfo: Added input-0-1434628147000 in memory on localhost:49443 (size: 2.4 KB, free: 247.3 MB)
15/06/18 13:49:07 INFO BlockGenerator: Pushed block input-0-1434628147000
15/06/18 13:49:10 INFO JobScheduler: Added jobs for time 1434628150000 ms
15/06/18 13:49:20 INFO JobScheduler: Added jobs for time 1434628160000 ms
{quote}
any ideas what i'm doing wrong? thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org