You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hafsa Asif <ha...@matchinguu.com> on 2015/07/07 12:38:07 UTC
How to solve ThreadException in Apache Spark standalone Java
Application
Hi,
I run the following simple Java spark standalone app with maven command
"exec:java -Dexec.mainClass=SimpleApp"
public class SimpleApp {
public static void main(String[] args) {
System.out.println("Reading and Connecting with Spark.....");
try {
String logFile = "/home/asif/spark-1.4.0/README.md"; // Should
be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple
Application").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> logData = sc.textFile(logFile).cache();
long numAs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("b"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with b: "
+ numBs);
}
catch(Exception e){
System.out.println ("Error in connecting with Spark");
}
}
}
Well, it builds successfully and also giving results but with thread
exception. What is the reason of the thread exception and how to solve it
in standalone mode because in spark shell with spark commit command, it is
running fine.
Log trace is:
[INFO] Scanning for projects...
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building standAloneSparkApp 1.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ standAloneSparkApp
---
Reading and Connecting with Spark.....
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/07/07 03:28:34 INFO SparkContext: Running Spark version 1.4.0
15/07/07 03:28:34 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/07/07 03:28:34 WARN Utils: Your hostname, ubuntu resolves to a loopback
address: 127.0.1.1; using 192.168.116.133 instead (on interface eth0)
15/07/07 03:28:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
15/07/07 03:28:34 INFO SecurityManager: Changing view acls to: asif
15/07/07 03:28:34 INFO SecurityManager: Changing modify acls to: asif
15/07/07 03:28:34 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(asif); users
with modify permissions: Set(asif)
15/07/07 03:28:35 INFO Slf4jLogger: Slf4jLogger started
15/07/07 03:28:36 INFO Remoting: Starting remoting
15/07/07 03:28:36 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@192.168.116.133:34863]
15/07/07 03:28:36 INFO Utils: Successfully started service 'sparkDriver' on
port 34863.
15/07/07 03:28:36 INFO SparkEnv: Registering MapOutputTracker
15/07/07 03:28:36 INFO SparkEnv: Registering BlockManagerMaster
15/07/07 03:28:36 INFO DiskBlockManager: Created local directory at
/tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28
15/07/07 03:28:36 INFO MemoryStore: MemoryStore started with capacity 534.5
MB
15/07/07 03:28:36 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/httpd-6f024c0f-60c4-413b-bb29-eee93e697651
15/07/07 03:28:36 INFO HttpServer: Starting HTTP Server
15/07/07 03:28:36 INFO Utils: Successfully started service 'HTTP file
server' on port 46189.
15/07/07 03:28:37 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/07 03:28:37 INFO Utils: Successfully started service 'SparkUI' on port
4040.
15/07/07 03:28:37 INFO SparkUI: Started SparkUI at
http://192.168.116.133:4040
15/07/07 03:28:37 INFO Executor: Starting executor ID driver on host
localhost
15/07/07 03:28:38 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 36884.
15/07/07 03:28:38 INFO NettyBlockTransferService: Server created on 36884
15/07/07 03:28:38 INFO BlockManagerMaster: Trying to register BlockManager
15/07/07 03:28:38 INFO BlockManagerMasterEndpoint: Registering block manager
localhost:36884 with 534.5 MB RAM, BlockManagerId(driver, localhost, 36884)
15/07/07 03:28:38 INFO BlockManagerMaster: Registered BlockManager
15/07/07 03:28:39 INFO MemoryStore: ensureFreeSpace(110248) called with
curMem=0, maxMem=560497950
15/07/07 03:28:39 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 107.7 KB, free 534.4 MB)
15/07/07 03:28:40 INFO MemoryStore: ensureFreeSpace(10090) called with
curMem=110248, maxMem=560497950
15/07/07 03:28:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes
in memory (estimated size 9.9 KB, free 534.4 MB)
15/07/07 03:28:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on localhost:36884 (size: 9.9 KB, free: 534.5 MB)
15/07/07 03:28:40 INFO SparkContext: Created broadcast 0 from textFile at
SimpleApp.java:19
15/07/07 03:28:40 INFO FileInputFormat: Total input paths to process : 1
15/07/07 03:28:40 INFO SparkContext: Starting job: count at
SimpleApp.java:23
15/07/07 03:28:40 INFO DAGScheduler: Got job 0 (count at SimpleApp.java:23)
with 1 output partitions (allowLocal=false)
15/07/07 03:28:40 INFO DAGScheduler: Final stage: ResultStage 0(count at
SimpleApp.java:23)
15/07/07 03:28:40 INFO DAGScheduler: Parents of final stage: List()
15/07/07 03:28:40 INFO DAGScheduler: Missing parents: List()
15/07/07 03:28:40 INFO DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[2] at filter at SimpleApp.java:21), which has no missing
parents
15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
curMem=120338, maxMem=560497950
15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1 stored as values in
memory (estimated size 3.2 KB, free 534.4 MB)
15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
curMem=123618, maxMem=560497950
15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes
in memory (estimated size 1933.0 B, free 534.4 MB)
15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
15/07/07 03:28:41 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:874
15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 0 (MapPartitionsRDD[2] at filter at SimpleApp.java:21)
15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, localhost, PROCESS_LOCAL, 1410 bytes)
15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/07/07 03:28:41 INFO CacheManager: Partition rdd_1_0 not found, computing
it
15/07/07 03:28:41 INFO HadoopRDD: Input split:
file:/home/asif/spark-1.4.0/README.md:0+3624
15/07/07 03:28:41 INFO deprecation: mapred.tip.id is deprecated. Instead,
use mapreduce.task.id
15/07/07 03:28:41 INFO deprecation: mapred.task.id is deprecated. Instead,
use mapreduce.task.attempt.id
15/07/07 03:28:41 INFO deprecation: mapred.task.is.map is deprecated.
Instead, use mapreduce.task.ismap
15/07/07 03:28:41 INFO deprecation: mapred.task.partition is deprecated.
Instead, use mapreduce.task.partition
15/07/07 03:28:41 INFO deprecation: mapred.job.id is deprecated. Instead,
use mapreduce.job.id
15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(11576) called with
curMem=125551, maxMem=560497950
15/07/07 03:28:41 INFO MemoryStore: Block rdd_1_0 stored as values in memory
(estimated size 11.3 KB, free 534.4 MB)
15/07/07 03:28:41 INFO BlockManagerInfo: Added rdd_1_0 in memory on
localhost:36884 (size: 11.3 KB, free: 534.5 MB)
15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0).
2410 bytes result sent to driver
15/07/07 03:28:41 INFO DAGScheduler: ResultStage 0 (count at
SimpleApp.java:23) finished in 0.542 s
15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID
0) in 520 ms on localhost (1/1)
15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
15/07/07 03:28:41 INFO DAGScheduler: Job 0 finished: count at
SimpleApp.java:23, took 0.918158 s
15/07/07 03:28:41 INFO SparkContext: Starting job: count at
SimpleApp.java:27
15/07/07 03:28:41 INFO DAGScheduler: Got job 1 (count at SimpleApp.java:27)
with 1 output partitions (allowLocal=false)
15/07/07 03:28:41 INFO DAGScheduler: Final stage: ResultStage 1(count at
SimpleApp.java:27)
15/07/07 03:28:41 INFO DAGScheduler: Parents of final stage: List()
15/07/07 03:28:41 INFO DAGScheduler: Missing parents: List()
15/07/07 03:28:41 INFO DAGScheduler: Submitting ResultStage 1
(MapPartitionsRDD[3] at filter at SimpleApp.java:25), which has no missing
parents
15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
curMem=137127, maxMem=560497950
15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2 stored as values in
memory (estimated size 3.2 KB, free 534.4 MB)
15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
curMem=140407, maxMem=560497950
15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes
in memory (estimated size 1933.0 B, free 534.4 MB)
15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
15/07/07 03:28:41 INFO SparkContext: Created broadcast 2 from broadcast at
DAGScheduler.scala:874
15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 1 (MapPartitionsRDD[3] at filter at SimpleApp.java:25)
15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID
1, localhost, PROCESS_LOCAL, 1410 bytes)
15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
15/07/07 03:28:41 INFO BlockManager: Found block rdd_1_0 locally
15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1).
1830 bytes result sent to driver
15/07/07 03:28:41 INFO DAGScheduler: ResultStage 1 (count at
SimpleApp.java:27) finished in 0.020 s
15/07/07 03:28:41 INFO DAGScheduler: Job 1 finished: count at
SimpleApp.java:27, took 0.070602 s
Lines with a: 60, lines with b: 29
15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID
1) in 24 ms on localhost (1/1)
15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks
have all completed, from pool
15/07/07 03:28:41 ERROR ContextCleaner: Error in cleaning thread
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
at
org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:157)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
at
org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:154)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:67)
15/07/07 03:28:41 ERROR Utils: uncaught error in thread SparkListenerBus,
stopping SparkContext
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:65)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
15/07/07 03:28:42 INFO SparkUI: Stopped Spark web UI at
http://192.168.116.133:4040
15/07/07 03:28:42 INFO DAGScheduler: Stopping DAGScheduler
[WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] was interrupted
but is still alive after waiting at least 14999msecs
[WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.actor.default-dispatcher-3,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.actor.default-dispatcher-4,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.actor.default-dispatcher-5,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.actor.default-dispatcher-6,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.remote.default-remote-dispatcher-7,5,SimpleApp] will
linger despite being asked to die via interruption
[WARNING] thread Thread[New I/O worker #1,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[New I/O worker #2,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[New I/O boss #3,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[New I/O worker #4,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[New I/O worker #5,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[New I/O server boss #6,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread
Thread[sparkDriver-akka.remote.default-remote-dispatcher-15,5,SimpleApp]
will linger despite being asked to die via interruption
[WARNING] thread Thread[MAP_OUTPUT_TRACKER cleanup timer,5,SimpleApp] will
linger despite being asked to die via interruption
[WARNING] thread Thread[BLOCK_MANAGER cleanup timer,5,SimpleApp] will linger
despite being asked to die via interruption
[WARNING] thread Thread[BROADCAST_VARS cleanup timer,5,SimpleApp] will
linger despite being asked to die via interruption
[WARNING] thread Thread[qtp1611265991-33 Acceptor0
SocketConnector@0.0.0.0:46189,5,SimpleApp] will linger despite being asked
to die via interruption
[WARNING] thread Thread[heartbeat-receiver-event-loop-thread,5,SimpleApp]
will linger despite being asked to die via interruption
[WARNING] thread Thread[shuffle-server-0,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] thread Thread[SparkListenerBus,5,SimpleApp] will linger despite
being asked to die via interruption
[WARNING] NOTE: 20 thread(s) did not finish despite being asked to via
interruption. This is not a problem with exec:java, it is a problem with the
running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup
org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=SimpleApp,maxpri=10]
java.lang.IllegalThreadStateException
at java.lang.ThreadGroup.destroy(ThreadGroup.java:778)
at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 26.998s
[INFO] Finished at: Tue Jul 07 03:28:56 PDT 2015
[INFO] Final Memory: 20M/60M
[INFO]
------------------------------------------------------------------------
15/07/07 03:28:57 INFO DiskBlockManager: Shutdown hook called
15/07/07 03:28:57 INFO Utils: path =
/tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28,
already present as root for deletion.
15/07/07 03:28:57 INFO Utils: Shutdown hook called
15/07/07 03:28:57 INFO Utils: Deleting directory
/tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/userFiles-3d0c9f72-b6f0-431f-9ca4-c82d040411c7
15/07/07 03:28:57 INFO Utils: Deleting directory
/tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca
Process finished with exit code 0
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-solve-ThreadException-in-Apache-Spark-standalone-Java-Application-tp23671.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to solve ThreadException in Apache Spark standalone Java Application
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you try adding sc.stop at the end of your program? looks like its
having a hard-time closing off sparkcontext.
Thanks
Best Regards
On Tue, Jul 7, 2015 at 4:08 PM, Hafsa Asif <ha...@matchinguu.com>
wrote:
> Hi,
>
> I run the following simple Java spark standalone app with maven command
> "exec:java -Dexec.mainClass=SimpleApp"
>
> public class SimpleApp {
> public static void main(String[] args) {
> System.out.println("Reading and Connecting with Spark.....");
> try {
> String logFile = "/home/asif/spark-1.4.0/README.md"; // Should
> be some file on your system
> SparkConf conf = new SparkConf().setAppName("Simple
> Application").setMaster("local");
> JavaSparkContext sc = new JavaSparkContext(conf);
> JavaRDD<String> logData = sc.textFile(logFile).cache();
>
> long numAs = logData.filter(new Function<String, Boolean>() {
> public Boolean call(String s) { return s.contains("a"); }
> }).count();
>
> long numBs = logData.filter(new Function<String, Boolean>() {
> public Boolean call(String s) { return s.contains("b"); }
> }).count();
>
> System.out.println("Lines with a: " + numAs + ", lines with b:
> "
> + numBs);
> }
> catch(Exception e){
> System.out.println ("Error in connecting with Spark");
> }
> }
> }
>
> Well, it builds successfully and also giving results but with thread
> exception. What is the reason of the thread exception and how to solve it
> in standalone mode because in spark shell with spark commit command, it is
> running fine.
> Log trace is:
>
> [INFO] Scanning for projects...
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building standAloneSparkApp 1.0-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO]
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ standAloneSparkApp
> ---
> Reading and Connecting with Spark.....
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/07/07 03:28:34 INFO SparkContext: Running Spark version 1.4.0
> 15/07/07 03:28:34 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/07/07 03:28:34 WARN Utils: Your hostname, ubuntu resolves to a loopback
> address: 127.0.1.1; using 192.168.116.133 instead (on interface eth0)
> 15/07/07 03:28:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> 15/07/07 03:28:34 INFO SecurityManager: Changing view acls to: asif
> 15/07/07 03:28:34 INFO SecurityManager: Changing modify acls to: asif
> 15/07/07 03:28:34 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(asif); users
> with modify permissions: Set(asif)
> 15/07/07 03:28:35 INFO Slf4jLogger: Slf4jLogger started
> 15/07/07 03:28:36 INFO Remoting: Starting remoting
> 15/07/07 03:28:36 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@192.168.116.133:34863]
> 15/07/07 03:28:36 INFO Utils: Successfully started service 'sparkDriver' on
> port 34863.
> 15/07/07 03:28:36 INFO SparkEnv: Registering MapOutputTracker
> 15/07/07 03:28:36 INFO SparkEnv: Registering BlockManagerMaster
> 15/07/07 03:28:36 INFO DiskBlockManager: Created local directory at
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28
> 15/07/07 03:28:36 INFO MemoryStore: MemoryStore started with capacity 534.5
> MB
> 15/07/07 03:28:36 INFO HttpFileServer: HTTP File server directory is
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/httpd-6f024c0f-60c4-413b-bb29-eee93e697651
> 15/07/07 03:28:36 INFO HttpServer: Starting HTTP Server
> 15/07/07 03:28:36 INFO Utils: Successfully started service 'HTTP file
> server' on port 46189.
> 15/07/07 03:28:37 INFO SparkEnv: Registering OutputCommitCoordinator
> 15/07/07 03:28:37 INFO Utils: Successfully started service 'SparkUI' on
> port
> 4040.
> 15/07/07 03:28:37 INFO SparkUI: Started SparkUI at
> http://192.168.116.133:4040
> 15/07/07 03:28:37 INFO Executor: Starting executor ID driver on host
> localhost
> 15/07/07 03:28:38 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36884.
> 15/07/07 03:28:38 INFO NettyBlockTransferService: Server created on 36884
> 15/07/07 03:28:38 INFO BlockManagerMaster: Trying to register BlockManager
> 15/07/07 03:28:38 INFO BlockManagerMasterEndpoint: Registering block
> manager
> localhost:36884 with 534.5 MB RAM, BlockManagerId(driver, localhost, 36884)
> 15/07/07 03:28:38 INFO BlockManagerMaster: Registered BlockManager
> 15/07/07 03:28:39 INFO MemoryStore: ensureFreeSpace(110248) called with
> curMem=0, maxMem=560497950
> 15/07/07 03:28:39 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 107.7 KB, free 534.4 MB)
> 15/07/07 03:28:40 INFO MemoryStore: ensureFreeSpace(10090) called with
> curMem=110248, maxMem=560497950
> 15/07/07 03:28:40 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes
> in memory (estimated size 9.9 KB, free 534.4 MB)
> 15/07/07 03:28:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
> on localhost:36884 (size: 9.9 KB, free: 534.5 MB)
> 15/07/07 03:28:40 INFO SparkContext: Created broadcast 0 from textFile at
> SimpleApp.java:19
> 15/07/07 03:28:40 INFO FileInputFormat: Total input paths to process : 1
> 15/07/07 03:28:40 INFO SparkContext: Starting job: count at
> SimpleApp.java:23
> 15/07/07 03:28:40 INFO DAGScheduler: Got job 0 (count at SimpleApp.java:23)
> with 1 output partitions (allowLocal=false)
> 15/07/07 03:28:40 INFO DAGScheduler: Final stage: ResultStage 0(count at
> SimpleApp.java:23)
> 15/07/07 03:28:40 INFO DAGScheduler: Parents of final stage: List()
> 15/07/07 03:28:40 INFO DAGScheduler: Missing parents: List()
> 15/07/07 03:28:40 INFO DAGScheduler: Submitting ResultStage 0
> (MapPartitionsRDD[2] at filter at SimpleApp.java:21), which has no missing
> parents
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
> curMem=120338, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 3.2 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
> curMem=123618, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes
> in memory (estimated size 1933.0 B, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
> on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
> 15/07/07 03:28:41 INFO SparkContext: Created broadcast 1 from broadcast at
> DAGScheduler.scala:874
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
> ResultStage 0 (MapPartitionsRDD[2] at filter at SimpleApp.java:21)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
> 15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, localhost, PROCESS_LOCAL, 1410 bytes)
> 15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 15/07/07 03:28:41 INFO CacheManager: Partition rdd_1_0 not found, computing
> it
> 15/07/07 03:28:41 INFO HadoopRDD: Input split:
> file:/home/asif/spark-1.4.0/README.md:0+3624
> 15/07/07 03:28:41 INFO deprecation: mapred.tip.id is deprecated. Instead,
> use mapreduce.task.id
> 15/07/07 03:28:41 INFO deprecation: mapred.task.id is deprecated. Instead,
> use mapreduce.task.attempt.id
> 15/07/07 03:28:41 INFO deprecation: mapred.task.is.map is deprecated.
> Instead, use mapreduce.task.ismap
> 15/07/07 03:28:41 INFO deprecation: mapred.task.partition is deprecated.
> Instead, use mapreduce.task.partition
> 15/07/07 03:28:41 INFO deprecation: mapred.job.id is deprecated. Instead,
> use mapreduce.job.id
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(11576) called with
> curMem=125551, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block rdd_1_0 stored as values in
> memory
> (estimated size 11.3 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added rdd_1_0 in memory on
> localhost:36884 (size: 11.3 KB, free: 534.5 MB)
> 15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0).
> 2410 bytes result sent to driver
> 15/07/07 03:28:41 INFO DAGScheduler: ResultStage 0 (count at
> SimpleApp.java:23) finished in 0.542 s
> 15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID
> 0) in 520 ms on localhost (1/1)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 15/07/07 03:28:41 INFO DAGScheduler: Job 0 finished: count at
> SimpleApp.java:23, took 0.918158 s
> 15/07/07 03:28:41 INFO SparkContext: Starting job: count at
> SimpleApp.java:27
> 15/07/07 03:28:41 INFO DAGScheduler: Got job 1 (count at SimpleApp.java:27)
> with 1 output partitions (allowLocal=false)
> 15/07/07 03:28:41 INFO DAGScheduler: Final stage: ResultStage 1(count at
> SimpleApp.java:27)
> 15/07/07 03:28:41 INFO DAGScheduler: Parents of final stage: List()
> 15/07/07 03:28:41 INFO DAGScheduler: Missing parents: List()
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting ResultStage 1
> (MapPartitionsRDD[3] at filter at SimpleApp.java:25), which has no missing
> parents
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
> curMem=137127, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 3.2 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
> curMem=140407, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2_piece0 stored as
> bytes
> in memory (estimated size 1933.0 B, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
> on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
> 15/07/07 03:28:41 INFO SparkContext: Created broadcast 2 from broadcast at
> DAGScheduler.scala:874
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
> ResultStage 1 (MapPartitionsRDD[3] at filter at SimpleApp.java:25)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
> 15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID
> 1, localhost, PROCESS_LOCAL, 1410 bytes)
> 15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
> 15/07/07 03:28:41 INFO BlockManager: Found block rdd_1_0 locally
> 15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1).
> 1830 bytes result sent to driver
> 15/07/07 03:28:41 INFO DAGScheduler: ResultStage 1 (count at
> SimpleApp.java:27) finished in 0.020 s
> 15/07/07 03:28:41 INFO DAGScheduler: Job 1 finished: count at
> SimpleApp.java:27, took 0.070602 s
> Lines with a: 60, lines with b: 29
> 15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID
> 1) in 24 ms on localhost (1/1)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks
> have all completed, from pool
> 15/07/07 03:28:41 ERROR ContextCleaner: Error in cleaning thread
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> at
>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:157)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
> at
> org.apache.spark.ContextCleaner.org
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:154)
> at
> org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:67)
> 15/07/07 03:28:41 ERROR Utils: uncaught error in thread SparkListenerBus,
> stopping SparkContext
> java.lang.InterruptedException
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:65)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
> at
>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
> 15/07/07 03:28:42 INFO SparkUI: Stopped Spark web UI at
> http://192.168.116.133:4040
> 15/07/07 03:28:42 INFO DAGScheduler: Stopping DAGScheduler
> [WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] was
> interrupted
> but is still alive after waiting at least 14999msecs
> [WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-3,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-4,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-5,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-6,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.remote.default-remote-dispatcher-7,5,SimpleApp]
> will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #1,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #2,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O boss #3,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #4,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #5,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O server boss #6,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.remote.default-remote-dispatcher-15,5,SimpleApp]
> will linger despite being asked to die via interruption
> [WARNING] thread Thread[MAP_OUTPUT_TRACKER cleanup timer,5,SimpleApp] will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[BLOCK_MANAGER cleanup timer,5,SimpleApp] will
> linger
> despite being asked to die via interruption
> [WARNING] thread Thread[BROADCAST_VARS cleanup timer,5,SimpleApp] will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[qtp1611265991-33 Acceptor0
> SocketConnector@0.0.0.0:46189,5,SimpleApp] will linger despite being asked
> to die via interruption
> [WARNING] thread Thread[heartbeat-receiver-event-loop-thread,5,SimpleApp]
> will linger despite being asked to die via interruption
> [WARNING] thread Thread[shuffle-server-0,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[SparkListenerBus,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] NOTE: 20 thread(s) did not finish despite being asked to via
> interruption. This is not a problem with exec:java, it is a problem with
> the
> running code. Although not serious, it should be remedied.
> [WARNING] Couldn't destroy threadgroup
>
> org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=SimpleApp,maxpri=10]
> java.lang.IllegalThreadStateException
> at java.lang.ThreadGroup.destroy(ThreadGroup.java:778)
> at
> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
> at
>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
>
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at
>
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 26.998s
> [INFO] Finished at: Tue Jul 07 03:28:56 PDT 2015
> [INFO] Final Memory: 20M/60M
> [INFO]
> ------------------------------------------------------------------------
> 15/07/07 03:28:57 INFO DiskBlockManager: Shutdown hook called
> 15/07/07 03:28:57 INFO Utils: path =
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28,
> already present as root for deletion.
> 15/07/07 03:28:57 INFO Utils: Shutdown hook called
> 15/07/07 03:28:57 INFO Utils: Deleting directory
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/userFiles-3d0c9f72-b6f0-431f-9ca4-c82d040411c7
> 15/07/07 03:28:57 INFO Utils: Deleting directory
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca
>
> Process finished with exit code 0
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-solve-ThreadException-in-Apache-Spark-standalone-Java-Application-tp23671.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>