You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vr spark <vr...@gmail.com> on 2016/09/25 06:36:34 UTC
spark-submit failing but job running from scala ide
Hi,
I have this simple scala app which works fine when i run it as scala
application from the scala IDE for eclipse.
But when i export is as jar and run it from spark-submit i am getting below
error. Please suggest
*bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar*
16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
4040. Attempting port 4041.
16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI' on
port 4041.
16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://192.168.1.3:4041
16/09/24 23:15:15 INFO SparkContext: Added JAR
file:/Users/vr/Downloads/spark-2.0.0/test.jar at spark://
192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
localhost
16/09/24 23:15:15 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 59264.
16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
192.168.1.3:59264
16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID
1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/09/24 23:15:16 INFO Executor: Fetching spark://
192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
16/09/24 23:16:31 INFO Executor: Fetching spark://
192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
java.io.IOException: Failed to connect to /192.168.1.3:59263
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at
org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.executor.Executor.org
$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
*My Scala code*
package com.x.y.vr.spark.first
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp {
def main(args: Array[String]) {
val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md" //
Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext("local[*]", "RatingsCounter")
//val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}
Re: spark-submit failing but job running from scala ide
Posted by Marco Mistroni <mm...@gmail.com>.
Hi Vr
your code works fine for me, running on Windows 10 vs Spark 1.6.1
i m guessing your Spark installation could be busted?
That would explain why it works on your IDE, as you are just importing jars
in your project.
The java.io.IOException: Failed to connect to error is misleading, i have
seen similar error for two or three completely different usecases........
I'd suggest your either try to
- move down to Spark 1.4.0 or 1.5.2 (there are subtle differences between
these old version and spark 1.6.1
- or reinstall Spark 1.6.1 and start from running spark-examples via
spark-submit
- run spark-shell and enter your SimpleApp line by line, to see if you can
get better debugging infos
hth
marco.
On Mon, Sep 26, 2016 at 5:22 PM, vr spark <vr...@gmail.com> wrote:
> Hi Jacek/All,
>
> I restarted my terminal and then i try spark-submit and again getting
> those errors. How do i see how many "runtimes" are running and how to have
> only one? some how my spark 1.6 and spark 2.0 are conflicting. how to fix
> it?
>
> i installed spark 1.6 earlier using this steps http://genomegeek.
> blogspot.com/2014/11/how-to-install-apache-spark-on-mac-os-x.html
> i installed spark 2.0 using these steps http://blog.weetech.co/
> 2015/08/light-learning-apache-spark.html
>
> Here is the for run-example
>
> m-C02KL0B1FFT4:bin vr$ ./run-example SparkPi
> Using Spark's default log4j profile: org/apache/spark/log4j-
> defaults.properties
> 16/09/26 09:11:00 INFO SparkContext: Running Spark version 2.0.0
> 16/09/26 09:11:00 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/09/26 09:11:00 INFO SecurityManager: Changing view acls to: vr
> 16/09/26 09:11:00 INFO SecurityManager: Changing modify acls to: vr
> 16/09/26 09:11:00 INFO SecurityManager: Changing view acls groups to:
> 16/09/26 09:11:00 INFO SecurityManager: Changing modify acls groups to:
> 16/09/26 09:11:00 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vr); groups
> with view permissions: Set(); users with modify permissions: Set(vr);
> groups with modify permissions: Set()
> 16/09/26 09:11:01 INFO Utils: Successfully started service 'sparkDriver'
> on port 59323.
> 16/09/26 09:11:01 INFO SparkEnv: Registering MapOutputTracker
> 16/09/26 09:11:01 INFO SparkEnv: Registering BlockManagerMaster
> 16/09/26 09:11:01 INFO DiskBlockManager: Created local directory at
> /private/var/folders/23/ycbtxh8s551gzlsgj8q647d88gsjgb
> /T/blockmgr-d0d6dfea-2c97-4337-8e7d-0bbcb141f4c9
> 16/09/26 09:11:01 INFO MemoryStore: MemoryStore started with capacity
> 366.3 MB
> 16/09/26 09:11:01 INFO SparkEnv: Registering OutputCommitCoordinator
> 16/09/26 09:11:01 WARN Utils: Service 'SparkUI' could not bind on port
> 4040. Attempting port 4041.
> 16/09/26 09:11:01 INFO Utils: Successfully started service 'SparkUI' on
> port 4041.
> 16/09/26 09:11:01 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.1.3:4041
> 16/09/26 09:11:01 INFO SparkContext: Added JAR file:/Users/vr/Downloads/
> spark-2.0.0/examples/target/scala-2.11/jars/scopt_2.11-3.3.0.jar at
> spark://192.168.1.3:59323/jars/scopt_2.11-3.3.0.jar with timestamp
> 1474906261472
> 16/09/26 09:11:01 INFO SparkContext: Added JAR file:/Users/vr/Downloads/
> spark-2.0.0/examples/target/scala-2.11/jars/spark-examples_2.11-2.0.0.jar
> at spark://192.168.1.3:59323/jars/spark-examples_2.11-2.0.0.jar with
> timestamp 1474906261473
> 16/09/26 09:11:01 INFO Executor: Starting executor ID driver on host
> localhost
> 16/09/26 09:11:01 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59324.
> 16/09/26 09:11:01 INFO NettyBlockTransferService: Server created on
> 192.168.1.3:59324
> 16/09/26 09:11:01 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, 192.168.1.3, 59324)
> 16/09/26 09:11:01 INFO BlockManagerMasterEndpoint: Registering block
> manager 192.168.1.3:59324 with 366.3 MB RAM, BlockManagerId(driver,
> 192.168.1.3, 59324)
> 16/09/26 09:11:01 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, 192.168.1.3, 59324)
> 16/09/26 09:11:01 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> 16/09/26 09:11:01 INFO SharedState: Warehouse path is
> 'file:/Users/vr/Downloads/spark-2.0.0/bin/spark-warehouse'.
> 16/09/26 09:11:01 INFO SparkContext: Starting job: reduce at
> SparkPi.scala:38
> 16/09/26 09:11:02 INFO DAGScheduler: Got job 0 (reduce at
> SparkPi.scala:38) with 2 output partitions
> 16/09/26 09:11:02 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at
> SparkPi.scala:38)
> 16/09/26 09:11:02 INFO DAGScheduler: Parents of final stage: List()
> 16/09/26 09:11:02 INFO DAGScheduler: Missing parents: List()
> 16/09/26 09:11:02 INFO DAGScheduler: Submitting ResultStage 0
> (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing
> parents
> 16/09/26 09:11:02 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 1832.0 B, free 366.3 MB)
> 16/09/26 09:11:02 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 1169.0 B, free 366.3 MB)
> 16/09/26 09:11:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on 192.168.1.3:59324 (size: 1169.0 B, free: 366.3 MB)
> 16/09/26 09:11:02 INFO SparkContext: Created broadcast 0 from broadcast at
> DAGScheduler.scala:1012
> 16/09/26 09:11:02 INFO DAGScheduler: Submitting 2 missing tasks from
> ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)
> 16/09/26 09:11:02 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
> 16/09/26 09:11:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, localhost, partition 0, PROCESS_LOCAL, 5474 bytes)
> 16/09/26 09:11:02 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID
> 1, localhost, partition 1, PROCESS_LOCAL, 5474 bytes)
> 16/09/26 09:11:02 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
> 16/09/26 09:11:02 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 16/09/26 09:11:02 INFO Executor: Fetching spark://192.168.1.3:59323/
> jars/scopt_2.11-3.3.0.jar with timestamp 1474906261472
> 16/09/26 09:12:17 INFO Executor: Fetching spark://192.168.1.3:59323/
> jars/scopt_2.11-3.3.0.jar with timestamp 1474906261472
> 16/09/26 09:12:17 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
> 0)
> java.io.IOException: Failed to connect to /192.168.1.3:59323
> java.io.IOException: Failed to connect to /192.168.1.3:59323
> at org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:228)
> at org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:179)
> at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(
> NettyRpcEnv.scala:358)
> at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(
> NettyRpcEnv.scala:324)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
> at org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
> at org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(
> TraversableLike.scala:733)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> at scala.collection.mutable.HashTable$class.foreachEntry(
> HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at scala.collection.TraversableLike$WithFilter.
> foreach(TraversableLike.scala:732)
> at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$
> updateDependencies(Executor.scala:480)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.ConnectException: Operation timed out: /
> 192.168.1.3:59323
>
>
>
>
>
> On Sun, Sep 25, 2016 at 8:32 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> How did you install Spark 1.6? It's usually as simple as rm -rf
>> $SPARK_1.6_HOME, but it really depends on how you installed it in the
>> first place.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Sun, Sep 25, 2016 at 4:32 PM, vr spark <vr...@gmail.com> wrote:
>> > yes, i have both spark 1.6 and spark 2.0.
>> > I unset the spark home environment variable and pointed spark submit to
>> 2.0.
>> > Its working now.
>> >
>> > How do i uninstall/remove spark 1.6 from mac?
>> >
>> > Thanks
>> >
>> >
>> > On Sun, Sep 25, 2016 at 4:28 AM, Jacek Laskowski <ja...@japila.pl>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Can you execute run-example SparkPi with your Spark installation?
>> >>
>> >> Also, see the logs:
>> >>
>> >> 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
>> >> 4040. Attempting port 4041.
>> >>
>> >> 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
>> >> on port 4041.
>> >>
>> >> You've got two Spark runtimes up that may or may not contribute to the
>> >> issue.
>> >>
>> >> Pozdrawiam,
>> >> Jacek Laskowski
>> >> ----
>> >> https://medium.com/@jaceklaskowski/
>> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> >> Follow me at https://twitter.com/jaceklaskowski
>> >>
>> >>
>> >> On Sun, Sep 25, 2016 at 8:36 AM, vr spark <vr...@gmail.com>
>> wrote:
>> >> > Hi,
>> >> > I have this simple scala app which works fine when i run it as scala
>> >> > application from the scala IDE for eclipse.
>> >> > But when i export is as jar and run it from spark-submit i am getting
>> >> > below
>> >> > error. Please suggest
>> >> >
>> >> > bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar
>> >> >
>> >> > 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on
>> port
>> >> > 4040.
>> >> > Attempting port 4041.
>> >> >
>> >> > 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
>> on
>> >> > port
>> >> > 4041.
>> >> >
>> >> > 16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and
>> started at
>> >> > http://192.168.1.3:4041
>> >> >
>> >> > 16/09/24 23:15:15 INFO SparkContext: Added JAR
>> >> > file:/Users/vr/Downloads/spark-2.0.0/test.jar at
>> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >> >
>> >> > 16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
>> >> > localhost
>> >> >
>> >> > 16/09/24 23:15:15 INFO Utils: Successfully started service
>> >> > 'org.apache.spark.network.netty.NettyBlockTransferService' on port
>> >> > 59264.
>> >> >
>> >> > 16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
>> >> > 192.168.1.3:59264
>> >> >
>> >> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0
>> >> > (TID
>> >> > 0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
>> >> >
>> >> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0
>> >> > (TID
>> >> > 1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
>> >> >
>> >> > 16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID
>> 0)
>> >> >
>> >> > 16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID
>> 1)
>> >> >
>> >> > 16/09/24 23:15:16 INFO Executor: Fetching
>> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >> >
>> >> > 16/09/24 23:16:31 INFO Executor: Fetching
>> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >> >
>> >> > 16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0
>> >> > (TID 1)
>> >> >
>> >> > java.io.IOException: Failed to connect to /192.168.1.3:59263
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.network.client.TransportClientFactory.creat
>> eClient(TransportClientFactory.java:228)
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.network.client.TransportClientFactory.creat
>> eClient(TransportClientFactory.java:179)
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyR
>> pcEnv.scala:358)
>> >> >
>> >> > at
>> >> > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcE
>> nv.scala:324)
>> >> >
>> >> > at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
>> >> >
>> >> > at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.executor.Executor$$anonfun$org$apache$spark
>> $executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.executor.Executor$$anonfun$org$apache$spark
>> $executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
>> >> >
>> >> > at
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter$$anonfun$
>> foreach$1.apply(TraversableLike.scala:733)
>> >> >
>> >> > at
>> >> >
>> >> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(
>> HashMap.scala:99)
>> >> >
>> >> > at
>> >> >
>> >> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(
>> HashMap.scala:99)
>> >> >
>> >> > at
>> >> >
>> >> > scala.collection.mutable.HashTable$class.foreachEntry(HashTa
>> ble.scala:230)
>> >> >
>> >> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>> >> >
>> >> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
>> >> >
>> >> > at
>> >> >
>> >> > scala.collection.TraversableLike$WithFilter.foreach(
>> TraversableLike.scala:732)
>> >> >
>> >> > at
>> >> >
>> >> > org.apache.spark.executor.Executor.org$apache$spark$executor
>> $Executor$$updateDependencies(Executor.scala:480)
>> >> >
>> >> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.
>> scala:252)
>> >> >
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> >> >
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:745)
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > My Scala code
>> >> >
>> >> >
>> >> > package com.x.y.vr.spark.first
>> >> >
>> >> > /* SimpleApp.scala */
>> >> >
>> >> > import org.apache.spark.SparkContext
>> >> >
>> >> > import org.apache.spark.SparkContext._
>> >> >
>> >> > import org.apache.spark.SparkConf
>> >> >
>> >> > object SimpleApp {
>> >> >
>> >> > def main(args: Array[String]) {
>> >> >
>> >> > val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md"
>> //
>> >> > Should
>> >> > be some file on your system
>> >> >
>> >> > val conf = new SparkConf().setAppName("Simple Application")
>> >> >
>> >> > val sc = new SparkContext("local[*]", "RatingsCounter")
>> >> >
>> >> > //val sc = new SparkContext(conf)
>> >> >
>> >> > val logData = sc.textFile(logFile, 2).cache()
>> >> >
>> >> > val numAs = logData.filter(line => line.contains("a")).count()
>> >> >
>> >> > val numBs = logData.filter(line => line.contains("b")).count()
>> >> >
>> >> > println("Lines with a: %s, Lines with b: %s".format(numAs,
>> numBs))
>> >> >
>> >> > }
>> >> >
>> >> > }
>> >
>> >
>>
>
>
Re: spark-submit failing but job running from scala ide
Posted by vr spark <vr...@gmail.com>.
Hi Jacek/All,
I restarted my terminal and then i try spark-submit and again getting
those errors. How do i see how many "runtimes" are running and how to have
only one? some how my spark 1.6 and spark 2.0 are conflicting. how to fix
it?
i installed spark 1.6 earlier using this steps
http://genomegeek.blogspot.com/2014/11/how-to-install-apache-spark-on-mac-os-x.html
i installed spark 2.0 using these steps
http://blog.weetech.co/2015/08/light-learning-apache-spark.html
Here is the for run-example
m-C02KL0B1FFT4:bin vr$ ./run-example SparkPi
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
16/09/26 09:11:00 INFO SparkContext: Running Spark version 2.0.0
16/09/26 09:11:00 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/09/26 09:11:00 INFO SecurityManager: Changing view acls to: vr
16/09/26 09:11:00 INFO SecurityManager: Changing modify acls to: vr
16/09/26 09:11:00 INFO SecurityManager: Changing view acls groups to:
16/09/26 09:11:00 INFO SecurityManager: Changing modify acls groups to:
16/09/26 09:11:00 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(vr); groups
with view permissions: Set(); users with modify permissions: Set(vr);
groups with modify permissions: Set()
16/09/26 09:11:01 INFO Utils: Successfully started service 'sparkDriver' on
port 59323.
16/09/26 09:11:01 INFO SparkEnv: Registering MapOutputTracker
16/09/26 09:11:01 INFO SparkEnv: Registering BlockManagerMaster
16/09/26 09:11:01 INFO DiskBlockManager: Created local directory at
/private/var/folders/23/ycbtxh8s551gzlsgj8q647d88gsjgb/T/blockmgr-d0d6dfea-2c97-4337-8e7d-0bbcb141f4c9
16/09/26 09:11:01 INFO MemoryStore: MemoryStore started with capacity 366.3
MB
16/09/26 09:11:01 INFO SparkEnv: Registering OutputCommitCoordinator
16/09/26 09:11:01 WARN Utils: Service 'SparkUI' could not bind on port
4040. Attempting port 4041.
16/09/26 09:11:01 INFO Utils: Successfully started service 'SparkUI' on
port 4041.
16/09/26 09:11:01 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://192.168.1.3:4041
16/09/26 09:11:01 INFO SparkContext: Added JAR
file:/Users/vr/Downloads/spark-2.0.0/examples/target/scala-2.11/jars/scopt_2.11-3.3.0.jar
at spark://192.168.1.3:59323/jars/scopt_2.11-3.3.0.jar with timestamp
1474906261472
16/09/26 09:11:01 INFO SparkContext: Added JAR
file:/Users/vr/Downloads/spark-2.0.0/examples/target/scala-2.11/jars/spark-examples_2.11-2.0.0.jar
at spark://192.168.1.3:59323/jars/spark-examples_2.11-2.0.0.jar with
timestamp 1474906261473
16/09/26 09:11:01 INFO Executor: Starting executor ID driver on host
localhost
16/09/26 09:11:01 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 59324.
16/09/26 09:11:01 INFO NettyBlockTransferService: Server created on
192.168.1.3:59324
16/09/26 09:11:01 INFO BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, 192.168.1.3, 59324)
16/09/26 09:11:01 INFO BlockManagerMasterEndpoint: Registering block
manager 192.168.1.3:59324 with 366.3 MB RAM, BlockManagerId(driver,
192.168.1.3, 59324)
16/09/26 09:11:01 INFO BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, 192.168.1.3, 59324)
16/09/26 09:11:01 WARN SparkContext: Use an existing SparkContext, some
configuration may not take effect.
16/09/26 09:11:01 INFO SharedState: Warehouse path is
'file:/Users/vr/Downloads/spark-2.0.0/bin/spark-warehouse'.
16/09/26 09:11:01 INFO SparkContext: Starting job: reduce at
SparkPi.scala:38
16/09/26 09:11:02 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38)
with 2 output partitions
16/09/26 09:11:02 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at
SparkPi.scala:38)
16/09/26 09:11:02 INFO DAGScheduler: Parents of final stage: List()
16/09/26 09:11:02 INFO DAGScheduler: Missing parents: List()
16/09/26 09:11:02 INFO DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing
parents
16/09/26 09:11:02 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 1832.0 B, free 366.3 MB)
16/09/26 09:11:02 INFO MemoryStore: Block broadcast_0_piece0 stored as
bytes in memory (estimated size 1169.0 B, free 366.3 MB)
16/09/26 09:11:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 192.168.1.3:59324 (size: 1169.0 B, free: 366.3 MB)
16/09/26 09:11:02 INFO SparkContext: Created broadcast 0 from broadcast at
DAGScheduler.scala:1012
16/09/26 09:11:02 INFO DAGScheduler: Submitting 2 missing tasks from
ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)
16/09/26 09:11:02 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/09/26 09:11:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, localhost, partition 0, PROCESS_LOCAL, 5474 bytes)
16/09/26 09:11:02 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID
1, localhost, partition 1, PROCESS_LOCAL, 5474 bytes)
16/09/26 09:11:02 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/09/26 09:11:02 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/09/26 09:11:02 INFO Executor: Fetching spark://
192.168.1.3:59323/jars/scopt_2.11-3.3.0.jar with timestamp 1474906261472
16/09/26 09:12:17 INFO Executor: Fetching spark://
192.168.1.3:59323/jars/scopt_2.11-3.3.0.jar with timestamp 1474906261472
16/09/26 09:12:17 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: Failed to connect to /192.168.1.3:59323
java.io.IOException: Failed to connect to /192.168.1.3:59323
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at
org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.executor.Executor.org
$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Operation timed out: /
192.168.1.3:59323
On Sun, Sep 25, 2016 at 8:32 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> How did you install Spark 1.6? It's usually as simple as rm -rf
> $SPARK_1.6_HOME, but it really depends on how you installed it in the
> first place.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Sun, Sep 25, 2016 at 4:32 PM, vr spark <vr...@gmail.com> wrote:
> > yes, i have both spark 1.6 and spark 2.0.
> > I unset the spark home environment variable and pointed spark submit to
> 2.0.
> > Its working now.
> >
> > How do i uninstall/remove spark 1.6 from mac?
> >
> > Thanks
> >
> >
> > On Sun, Sep 25, 2016 at 4:28 AM, Jacek Laskowski <ja...@japila.pl>
> wrote:
> >>
> >> Hi,
> >>
> >> Can you execute run-example SparkPi with your Spark installation?
> >>
> >> Also, see the logs:
> >>
> >> 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
> >> 4040. Attempting port 4041.
> >>
> >> 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
> >> on port 4041.
> >>
> >> You've got two Spark runtimes up that may or may not contribute to the
> >> issue.
> >>
> >> Pozdrawiam,
> >> Jacek Laskowski
> >> ----
> >> https://medium.com/@jaceklaskowski/
> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> >> Follow me at https://twitter.com/jaceklaskowski
> >>
> >>
> >> On Sun, Sep 25, 2016 at 8:36 AM, vr spark <vr...@gmail.com> wrote:
> >> > Hi,
> >> > I have this simple scala app which works fine when i run it as scala
> >> > application from the scala IDE for eclipse.
> >> > But when i export is as jar and run it from spark-submit i am getting
> >> > below
> >> > error. Please suggest
> >> >
> >> > bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar
> >> >
> >> > 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
> >> > 4040.
> >> > Attempting port 4041.
> >> >
> >> > 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
> on
> >> > port
> >> > 4041.
> >> >
> >> > 16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started
> at
> >> > http://192.168.1.3:4041
> >> >
> >> > 16/09/24 23:15:15 INFO SparkContext: Added JAR
> >> > file:/Users/vr/Downloads/spark-2.0.0/test.jar at
> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >> >
> >> > 16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
> >> > localhost
> >> >
> >> > 16/09/24 23:15:15 INFO Utils: Successfully started service
> >> > 'org.apache.spark.network.netty.NettyBlockTransferService' on port
> >> > 59264.
> >> >
> >> > 16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
> >> > 192.168.1.3:59264
> >> >
> >> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0
> >> > (TID
> >> > 0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
> >> >
> >> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0
> >> > (TID
> >> > 1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
> >> >
> >> > 16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> >> >
> >> > 16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
> >> >
> >> > 16/09/24 23:15:16 INFO Executor: Fetching
> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >> >
> >> > 16/09/24 23:16:31 INFO Executor: Fetching
> >> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >> >
> >> > 16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0
> >> > (TID 1)
> >> >
> >> > java.io.IOException: Failed to connect to /192.168.1.3:59263
> >> >
> >> > at
> >> >
> >> > org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:228)
> >> >
> >> > at
> >> >
> >> > org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:179)
> >> >
> >> > at
> >> >
> >> > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(
> NettyRpcEnv.scala:358)
> >> >
> >> > at
> >> > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(
> NettyRpcEnv.scala:324)
> >> >
> >> > at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
> >> >
> >> > at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
> >> >
> >> > at
> >> >
> >> > org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
> >> >
> >> > at
> >> >
> >> > org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
> >> >
> >> > at
> >> >
> >> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(
> TraversableLike.scala:733)
> >> >
> >> > at
> >> >
> >> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> >> >
> >> > at
> >> >
> >> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> >> >
> >> > at
> >> >
> >> > scala.collection.mutable.HashTable$class.foreachEntry(
> HashTable.scala:230)
> >> >
> >> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> >> >
> >> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> >> >
> >> > at
> >> >
> >> > scala.collection.TraversableLike$WithFilter.
> foreach(TraversableLike.scala:732)
> >> >
> >> > at
> >> >
> >> > org.apache.spark.executor.Executor.org$apache$spark$
> executor$Executor$$updateDependencies(Executor.scala:480)
> >> >
> >> > at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:252)
> >> >
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >> >
> >> > at
> >> >
> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >> >
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > My Scala code
> >> >
> >> >
> >> > package com.x.y.vr.spark.first
> >> >
> >> > /* SimpleApp.scala */
> >> >
> >> > import org.apache.spark.SparkContext
> >> >
> >> > import org.apache.spark.SparkContext._
> >> >
> >> > import org.apache.spark.SparkConf
> >> >
> >> > object SimpleApp {
> >> >
> >> > def main(args: Array[String]) {
> >> >
> >> > val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md" //
> >> > Should
> >> > be some file on your system
> >> >
> >> > val conf = new SparkConf().setAppName("Simple Application")
> >> >
> >> > val sc = new SparkContext("local[*]", "RatingsCounter")
> >> >
> >> > //val sc = new SparkContext(conf)
> >> >
> >> > val logData = sc.textFile(logFile, 2).cache()
> >> >
> >> > val numAs = logData.filter(line => line.contains("a")).count()
> >> >
> >> > val numBs = logData.filter(line => line.contains("b")).count()
> >> >
> >> > println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
> >> >
> >> > }
> >> >
> >> > }
> >
> >
>
Re: spark-submit failing but job running from scala ide
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
How did you install Spark 1.6? It's usually as simple as rm -rf
$SPARK_1.6_HOME, but it really depends on how you installed it in the
first place.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sun, Sep 25, 2016 at 4:32 PM, vr spark <vr...@gmail.com> wrote:
> yes, i have both spark 1.6 and spark 2.0.
> I unset the spark home environment variable and pointed spark submit to 2.0.
> Its working now.
>
> How do i uninstall/remove spark 1.6 from mac?
>
> Thanks
>
>
> On Sun, Sep 25, 2016 at 4:28 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi,
>>
>> Can you execute run-example SparkPi with your Spark installation?
>>
>> Also, see the logs:
>>
>> 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
>> 4040. Attempting port 4041.
>>
>> 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
>> on port 4041.
>>
>> You've got two Spark runtimes up that may or may not contribute to the
>> issue.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Sun, Sep 25, 2016 at 8:36 AM, vr spark <vr...@gmail.com> wrote:
>> > Hi,
>> > I have this simple scala app which works fine when i run it as scala
>> > application from the scala IDE for eclipse.
>> > But when i export is as jar and run it from spark-submit i am getting
>> > below
>> > error. Please suggest
>> >
>> > bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar
>> >
>> > 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
>> > 4040.
>> > Attempting port 4041.
>> >
>> > 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI' on
>> > port
>> > 4041.
>> >
>> > 16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
>> > http://192.168.1.3:4041
>> >
>> > 16/09/24 23:15:15 INFO SparkContext: Added JAR
>> > file:/Users/vr/Downloads/spark-2.0.0/test.jar at
>> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >
>> > 16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
>> > localhost
>> >
>> > 16/09/24 23:15:15 INFO Utils: Successfully started service
>> > 'org.apache.spark.network.netty.NettyBlockTransferService' on port
>> > 59264.
>> >
>> > 16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
>> > 192.168.1.3:59264
>> >
>> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0
>> > (TID
>> > 0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
>> >
>> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0
>> > (TID
>> > 1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
>> >
>> > 16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
>> >
>> > 16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
>> >
>> > 16/09/24 23:15:16 INFO Executor: Fetching
>> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >
>> > 16/09/24 23:16:31 INFO Executor: Fetching
>> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>> >
>> > 16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0
>> > (TID 1)
>> >
>> > java.io.IOException: Failed to connect to /192.168.1.3:59263
>> >
>> > at
>> >
>> > org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
>> >
>> > at
>> >
>> > org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
>> >
>> > at
>> >
>> > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
>> >
>> > at
>> > org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
>> >
>> > at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
>> >
>> > at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
>> >
>> > at
>> >
>> > org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
>> >
>> > at
>> >
>> > org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
>> >
>> > at
>> >
>> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>> >
>> > at
>> >
>> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>> >
>> > at
>> >
>> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>> >
>> > at
>> >
>> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
>> >
>> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>> >
>> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
>> >
>> > at
>> >
>> > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
>> >
>> > at
>> >
>> > org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
>> >
>> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
>> >
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> >
>> >
>> >
>> >
>> > My Scala code
>> >
>> >
>> > package com.x.y.vr.spark.first
>> >
>> > /* SimpleApp.scala */
>> >
>> > import org.apache.spark.SparkContext
>> >
>> > import org.apache.spark.SparkContext._
>> >
>> > import org.apache.spark.SparkConf
>> >
>> > object SimpleApp {
>> >
>> > def main(args: Array[String]) {
>> >
>> > val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md" //
>> > Should
>> > be some file on your system
>> >
>> > val conf = new SparkConf().setAppName("Simple Application")
>> >
>> > val sc = new SparkContext("local[*]", "RatingsCounter")
>> >
>> > //val sc = new SparkContext(conf)
>> >
>> > val logData = sc.textFile(logFile, 2).cache()
>> >
>> > val numAs = logData.filter(line => line.contains("a")).count()
>> >
>> > val numBs = logData.filter(line => line.contains("b")).count()
>> >
>> > println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
>> >
>> > }
>> >
>> > }
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: spark-submit failing but job running from scala ide
Posted by vr spark <vr...@gmail.com>.
yes, i have both spark 1.6 and spark 2.0.
I unset the spark home environment variable and pointed spark submit to 2.0.
Its working now.
How do i uninstall/remove spark 1.6 from mac?
Thanks
On Sun, Sep 25, 2016 at 4:28 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Can you execute run-example SparkPi with your Spark installation?
>
> Also, see the logs:
>
> 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
> 4040. Attempting port 4041.
>
> 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
> on port 4041.
>
> You've got two Spark runtimes up that may or may not contribute to the
> issue.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Sun, Sep 25, 2016 at 8:36 AM, vr spark <vr...@gmail.com> wrote:
> > Hi,
> > I have this simple scala app which works fine when i run it as scala
> > application from the scala IDE for eclipse.
> > But when i export is as jar and run it from spark-submit i am getting
> below
> > error. Please suggest
> >
> > bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar
> >
> > 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
> 4040.
> > Attempting port 4041.
> >
> > 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI' on
> port
> > 4041.
> >
> > 16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> > http://192.168.1.3:4041
> >
> > 16/09/24 23:15:15 INFO SparkContext: Added JAR
> > file:/Users/vr/Downloads/spark-2.0.0/test.jar at
> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >
> > 16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
> > localhost
> >
> > 16/09/24 23:15:15 INFO Utils: Successfully started service
> > 'org.apache.spark.network.netty.NettyBlockTransferService' on port
> 59264.
> >
> > 16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
> > 192.168.1.3:59264
> >
> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0
> (TID
> > 0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
> >
> > 16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0
> (TID
> > 1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
> >
> > 16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> >
> > 16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
> >
> > 16/09/24 23:15:16 INFO Executor: Fetching
> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >
> > 16/09/24 23:16:31 INFO Executor: Fetching
> > spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
> >
> > 16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0
> (TID 1)
> >
> > java.io.IOException: Failed to connect to /192.168.1.3:59263
> >
> > at
> > org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:228)
> >
> > at
> > org.apache.spark.network.client.TransportClientFactory.createClient(
> TransportClientFactory.java:179)
> >
> > at
> > org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(
> NettyRpcEnv.scala:358)
> >
> > at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(
> NettyRpcEnv.scala:324)
> >
> > at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
> >
> > at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
> >
> > at
> > org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
> >
> > at
> > org.apache.spark.executor.Executor$$anonfun$org$apache$
> spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
> >
> > at
> > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(
> TraversableLike.scala:733)
> >
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> >
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:99)
> >
> > at
> > scala.collection.mutable.HashTable$class.foreachEntry(
> HashTable.scala:230)
> >
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> >
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> >
> > at
> > scala.collection.TraversableLike$WithFilter.
> foreach(TraversableLike.scala:732)
> >
> > at
> > org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$
> updateDependencies(Executor.scala:480)
> >
> > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> >
> > My Scala code
> >
> >
> > package com.x.y.vr.spark.first
> >
> > /* SimpleApp.scala */
> >
> > import org.apache.spark.SparkContext
> >
> > import org.apache.spark.SparkContext._
> >
> > import org.apache.spark.SparkConf
> >
> > object SimpleApp {
> >
> > def main(args: Array[String]) {
> >
> > val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md" //
> Should
> > be some file on your system
> >
> > val conf = new SparkConf().setAppName("Simple Application")
> >
> > val sc = new SparkContext("local[*]", "RatingsCounter")
> >
> > //val sc = new SparkContext(conf)
> >
> > val logData = sc.textFile(logFile, 2).cache()
> >
> > val numAs = logData.filter(line => line.contains("a")).count()
> >
> > val numBs = logData.filter(line => line.contains("b")).count()
> >
> > println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
> >
> > }
> >
> > }
>
Re: spark-submit failing but job running from scala ide
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Can you execute run-example SparkPi with your Spark installation?
Also, see the logs:
16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port
4040. Attempting port 4041.
16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI'
on port 4041.
You've got two Spark runtimes up that may or may not contribute to the issue.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sun, Sep 25, 2016 at 8:36 AM, vr spark <vr...@gmail.com> wrote:
> Hi,
> I have this simple scala app which works fine when i run it as scala
> application from the scala IDE for eclipse.
> But when i export is as jar and run it from spark-submit i am getting below
> error. Please suggest
>
> bin/spark-submit --class com.x.y.vr.spark.first.SimpleApp test.jar
>
> 16/09/24 23:15:15 WARN Utils: Service 'SparkUI' could not bind on port 4040.
> Attempting port 4041.
>
> 16/09/24 23:15:15 INFO Utils: Successfully started service 'SparkUI' on port
> 4041.
>
> 16/09/24 23:15:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.1.3:4041
>
> 16/09/24 23:15:15 INFO SparkContext: Added JAR
> file:/Users/vr/Downloads/spark-2.0.0/test.jar at
> spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>
> 16/09/24 23:15:15 INFO Executor: Starting executor ID driver on host
> localhost
>
> 16/09/24 23:15:15 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59264.
>
> 16/09/24 23:15:15 INFO NettyBlockTransferService: Server created on
> 192.168.1.3:59264
>
> 16/09/24 23:15:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, localhost, partition 0, PROCESS_LOCAL, 5354 bytes)
>
> 16/09/24 23:15:16 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID
> 1, localhost, partition 1, PROCESS_LOCAL, 5354 bytes)
>
> 16/09/24 23:15:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
>
> 16/09/24 23:15:16 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
>
> 16/09/24 23:15:16 INFO Executor: Fetching
> spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>
> 16/09/24 23:16:31 INFO Executor: Fetching
> spark://192.168.1.3:59263/jars/test.jar with timestamp 1474784115210
>
> 16/09/24 23:16:31 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
>
> java.io.IOException: Failed to connect to /192.168.1.3:59263
>
> at
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
>
> at
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
>
> at
> org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
>
> at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
>
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
>
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
>
> at
> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
>
> at
> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
>
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
>
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
>
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
>
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
>
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
>
> at
> org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> My Scala code
>
>
> package com.x.y.vr.spark.first
>
> /* SimpleApp.scala */
>
> import org.apache.spark.SparkContext
>
> import org.apache.spark.SparkContext._
>
> import org.apache.spark.SparkConf
>
> object SimpleApp {
>
> def main(args: Array[String]) {
>
> val logFile = "/Users/vttrich/Downloads/spark-2.0.0/README.md" // Should
> be some file on your system
>
> val conf = new SparkConf().setAppName("Simple Application")
>
> val sc = new SparkContext("local[*]", "RatingsCounter")
>
> //val sc = new SparkContext(conf)
>
> val logData = sc.textFile(logFile, 2).cache()
>
> val numAs = logData.filter(line => line.contains("a")).count()
>
> val numBs = logData.filter(line => line.contains("b")).count()
>
> println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
>
> }
>
> }
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org