You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kushal Chokhani <ku...@enlightedinc.com> on 2015/08/06 08:10:31 UTC

spark job not accepting resources from worker

Hi

I have a spark/cassandra setup where I am using a spark cassandra java 
connector to query on a table. So far, I have 1 spark master node (2 
cores) and 1 worker node (4 cores). Both of them have following 
spark-env.sh under conf/:

    |#!/usr/bin/env bash
    export SPARK_LOCAL_IP=127.0.0.1
    export SPARK_MASTER_IP="192.168.4.134"
    export SPARK_WORKER_MEMORY=1G
    export SPARK_EXECUTOR_MEMORY=2G

    |

I am using spark 1.4.1 along with cassandra 2.2.0. I have started my 
cassandra/spark setup. Created keyspace and table under cassandra and 
added some rows on table. Now I try to run following spark job using 
spark cassandra java connector:

|     SparkConf conf = new SparkConf();
     conf.setAppName("Testing");
     conf.setMaster("spark://192.168.4.134:7077");
     conf.set("spark.cassandra.connection.host", "192.168.4.129");
     conf.set("spark.logConf", "true");
     conf.set("spark.driver.maxResultSize", "50m");
     conf.set("spark.executor.memory", "200m");
     conf.set("spark.eventLog.enabled", "true");
     conf.set("spark.eventLog.dir", "/tmp/");
     conf.set("spark.executor.extraClassPath", "/home/enlighted/ebd.jar");
     conf.set("spark.cores.max", "1");
     JavaSparkContext sc = new JavaSparkContext(conf);


     JavaRDD<String> cassandraRowsRDD = CassandraJavaUtil.javaFunctions(sc).cassandraTable("testing", "ec")
     .map(new Function<CassandraRow, String>() {
         private static final long serialVersionUID = -6263533266898869895L;
         @Override
         public String call(CassandraRow cassandraRow) throws Exception {
             return cassandraRow.toString();
         }
     });
     System.out.println("Data as CassandraRows: \n" + StringUtils.join(cassandraRowsRDD.toArray(), "\n"));
     sc.close();|



This job is stuck with insufficient resources warning. Here are logs:

    1107 [main] INFO org.apache.spark.SparkContext  - Spark configuration:
    spark.app.name=Testing
    spark.cassandra.connection.host=192.168.4.129
    spark.cores.max=1
    spark.driver.maxResultSize=50m
    spark.eventLog.dir=/tmp/
    spark.eventLog.enabled=true
    spark.executor.extraClassPath=/home/enlighted/ebd.jar
    spark.executor.memory=200m
    spark.logConf=true
    spark.master=spark://192.168.4.134:7077
    1121 [main] INFO org.apache.spark.SecurityManager  - Changing view
    acls to: enlighted
    1122 [main] INFO org.apache.spark.SecurityManager  - Changing modify
    acls to: enlighted
    1123 [main] INFO org.apache.spark.SecurityManager  -
    SecurityManager: authentication disabled; ui acls disabled; users
    with view permissions: Set(enlighted); users with modify
    permissions: Set(enlighted)
    1767 [sparkDriver-akka.actor.default-dispatcher-4] INFO
    akka.event.slf4j.Slf4jLogger  - Slf4jLogger started
    1805 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting -
    Starting remoting
    1957 [main] INFO org.apache.spark.util.Utils  - Successfully started
    service 'sparkDriver' on port 54611.
    1958 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting -
    Remoting started; listening on addresses
    :[akka.tcp://sparkDriver@192.168.4.134:54611]
    1977 [main] INFO org.apache.spark.SparkEnv  - Registering
    MapOutputTracker
    1989 [main] INFO org.apache.spark.SparkEnv  - Registering
    BlockManagerMaster
    2007 [main] INFO org.apache.spark.storage.DiskBlockManager  -
    Created local directory at
    /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/blockmgr-3e3d54e7-16df-4e97-be48-b0c0fa0389e7
    2012 [main] INFO org.apache.spark.storage.MemoryStore  - MemoryStore
    started with capacity 456.0 MB
    2044 [main] INFO org.apache.spark.HttpFileServer  - HTTP File server
    directory is
    /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/httpd-64b4d92e-cde9-45fb-8b38-edc3cca3933c
    2046 [main] INFO org.apache.spark.HttpServer  - Starting HTTP Server
    2086 [main] INFO org.spark-project.jetty.server.Server  -
    jetty-8.y.z-SNAPSHOT
    2098 [main] INFO org.spark-project.jetty.server.AbstractConnector -
    Started SocketConnector@0.0.0.0:44884
    2099 [main] INFO org.apache.spark.util.Utils  - Successfully started
    service 'HTTP file server' on port 44884.
    2108 [main] INFO org.apache.spark.SparkEnv  - Registering
    OutputCommitCoordinator
    2297 [main] INFO org.spark-project.jetty.server.Server  -
    jetty-8.y.z-SNAPSHOT
    2317 [main] INFO org.spark-project.jetty.server.AbstractConnector -
    Started SelectChannelConnector@0.0.0.0:4040
    2318 [main] INFO org.apache.spark.util.Utils  - Successfully started
    service 'SparkUI' on port 4040.
    2320 [main] INFO org.apache.spark.ui.SparkUI  - Started SparkUI at
    http://192.168.4.134:4040
    2387 [sparkDriver-akka.actor.default-dispatcher-3] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Connecting
    to master akka.tcp://sparkMaster@192.168.4.134:7077/user/Master...
    2662 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    Connected to Spark cluster with app ID app-20150806054450-0001
    2680 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    added: app-20150806054450-0001/0 on
    worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566) with
    1 cores
    2682 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    Granted executor ID app-20150806054450-0001/0 on hostPort
    192.168.4.129:45566 with 1 cores, 200.0 MB RAM
    2696 [main] INFO org.apache.spark.util.Utils  - Successfully started
    service 'org.apache.spark.network.netty.NettyBlockTransferService'
    on port 49150.
    2696 [main] INFO
    org.apache.spark.network.netty.NettyBlockTransferService  - Server
    created on 49150
    2700 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    updated: app-20150806054450-0001/0 is now LOADING
    2706 [main] INFO org.apache.spark.storage.BlockManagerMaster  -
    Trying to register BlockManager
    2708 [sparkDriver-akka.actor.default-dispatcher-17] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    updated: app-20150806054450-0001/0 is now RUNNING
    2710 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.storage.BlockManagerMasterEndpoint  - Registering
    block manager 192.168.4.134:49150 with 456.0 MB RAM,
    BlockManagerId(driver, 192.168.4.134, 49150)
    2713 [main] INFO org.apache.spark.storage.BlockManagerMaster  -
    Registered BlockManager
    2922 [main] INFO org.apache.spark.scheduler.EventLoggingListener -
    Logging events to file:/tmp/app-20150806054450-0001
    2939 [main] INFO
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    SchedulerBackend is ready for scheduling beginning after reached
    minRegisteredResourcesRatio: 0.0
    3321 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
    host /192.168.4.129:9042 added
    3321 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
    host /192.168.4.130:9042 added
    3322 [main] INFO
    com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
    Added host 192.168.4.130 (DC1)
    3322 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
    host /192.168.4.131:9042 added
    3323 [main] INFO
    com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
    Added host 192.168.4.131 (DC1)
    3323 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
    host /192.168.4.132:9042 added
    3323 [main] INFO
    com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
    Added host 192.168.4.132 (DC1)
    3325 [main] INFO
    com.datastax.spark.connector.cql.CassandraConnector  - Connected to
    Cassandra cluster: enldbcluster
    3881 [main] INFO org.apache.spark.SparkContext  - Starting job:
    toArray at Start.java:85
    3898 [pool-18-thread-1] INFO
    com.datastax.spark.connector.cql.CassandraConnector  - Disconnected
    from Cassandra cluster: enldbcluster
    3901 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Got job 0 (toArray at
    Start.java:85) with 6 output partitions (allowLocal=false)
    3902 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage
    0(toArray at Start.java:85)
    3902 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Parents of final stage:
    List()
    3908 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
    3925 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 0
    (MapPartitionsRDD[1] at map at Start.java:77), which has no missing
    parents
    4002 [dag-scheduler-event-loop] INFO
    org.apache.spark.storage.MemoryStore  - ensureFreeSpace(7488) called
    with curMem=0, maxMem=478182113
    4004 [dag-scheduler-event-loop] INFO
    org.apache.spark.storage.MemoryStore  - Block broadcast_0 stored as
    values in memory (estimated size 7.3 KB, free 456.0 MB)
    4013 [dag-scheduler-event-loop] INFO
    org.apache.spark.storage.MemoryStore  - ensureFreeSpace(4015) called
    with curMem=7488, maxMem=478182113
    4013 [dag-scheduler-event-loop] INFO
    org.apache.spark.storage.MemoryStore  - Block broadcast_0_piece0
    stored as bytes in memory (estimated size 3.9 KB, free 456.0 MB)
    4015 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.storage.BlockManagerInfo  - Added
    broadcast_0_piece0 in memory on 192.168.4.134:49150 (size: 3.9 KB,
    free: 456.0 MB)
    4017 [dag-scheduler-event-loop] INFO org.apache.spark.SparkContext 
    - Created broadcast 0 from broadcast at DAGScheduler.scala:874
    4089 [dag-scheduler-event-loop] INFO
    com.datastax.driver.core.Cluster  - New Cassandra host
    /192.168.4.129:9042 added
    4089 [dag-scheduler-event-loop] INFO
    com.datastax.driver.core.Cluster  - New Cassandra host
    /192.168.4.130:9042 added
    4089 [dag-scheduler-event-loop] INFO
    com.datastax.driver.core.Cluster  - New Cassandra host
    /192.168.4.131:9042 added
    4089 [dag-scheduler-event-loop] INFO
    com.datastax.driver.core.Cluster  - New Cassandra host
    /192.168.4.132:9042 added
    4089 [dag-scheduler-event-loop] INFO
    com.datastax.spark.connector.cql.CassandraConnector  - Connected to
    Cassandra cluster: enldbcluster
    4394 [pool-18-thread-1] INFO
    com.datastax.spark.connector.cql.CassandraConnector  - Disconnected
    from Cassandra cluster: enldbcluster
    4806 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.DAGScheduler  - Submitting 6 missing
    tasks from ResultStage 0 (MapPartitionsRDD[1] at map at Start.java:77)
    4807 [dag-scheduler-event-loop] INFO
    org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 0.0
    with 6 tasks
    19822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    34822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    49822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    64822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    79822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    94822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources
    109822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    124822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    124963 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    updated: app-20150806054450-0001/0 is now EXITED (Command exited
    with code 1)
    124964 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    Executor app-20150806054450-0001/0 removed: Command exited with code 1
    124968 [sparkDriver-akka.actor.default-dispatcher-17] ERROR
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    Asked to remove non-existent executor 0
    124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    added: app-20150806054450-0001/1 on
    worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566) with
    1 cores
    124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend  -
    Granted executor ID app-20150806054450-0001/1 on hostPort
    192.168.4.129:45566 with 1 cores, 200.0 MB RAM
    124975 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    updated: app-20150806054450-0001/1 is now RUNNING
    125012 [sparkDriver-akka.actor.default-dispatcher-14] INFO
    org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
    updated: app-20150806054450-0001/1 is now LOADING
    139822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    154822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    169823 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    184822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources
    199822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
    - Initial job has not accepted any resources; check your cluster UI
    to ensure that workers are registered and have sufficient resources

||

Please find attached the spark master UI and pom.xml file with dependencies.

Can anyone please point out what could be an issue here.




Re: spark job not accepting resources from worker

Posted by Kushal Chokhani <ku...@enlightedinc.com>.
Figured out the root cause. Master was randomly assigning port to worker 
for communication. Because of the firewall on master, worker couldn't 
send out messages to master (maybe like resource details). Weird worker 
didn't even bother to throw any error also.

On 8/6/2015 3:24 PM, Kushal Chokhani wrote:
> Any inputs?
>
> In case of following message, is there a way to check which resources 
> is not sufficient through some logs?
>
>     [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl  -
>     Initial job has not accepted any resources; check your cluster UI
>     to ensure that workers are registered and have sufficient resources
>
> Regards.
>
> On 8/6/2015 11:40 AM, Kushal Chokhani wrote:
>> Hi
>>
>> I have a spark/cassandra setup where I am using a spark cassandra 
>> java connector to query on a table. So far, I have 1 spark master 
>> node (2 cores) and 1 worker node (4 cores). Both of them have 
>> following spark-env.sh under conf/:
>>
>>     |#!/usr/bin/env bash
>>     export SPARK_LOCAL_IP=127.0.0.1
>>     export SPARK_MASTER_IP="192.168.4.134"
>>     export SPARK_WORKER_MEMORY=1G
>>     export SPARK_EXECUTOR_MEMORY=2G
>>
>>     |
>>
>> I am using spark 1.4.1 along with cassandra 2.2.0. I have started my 
>> cassandra/spark setup. Created keyspace and table under cassandra and 
>> added some rows on table. Now I try to run following spark job using 
>> spark cassandra java connector:
>>
>> |     SparkConf conf = new SparkConf();
>>      conf.setAppName("Testing");
>>      conf.setMaster("spark://192.168.4.134:7077");
>>      conf.set("spark.cassandra.connection.host", "192.168.4.129");
>>      conf.set("spark.logConf", "true");
>>      conf.set("spark.driver.maxResultSize", "50m");
>>      conf.set("spark.executor.memory", "200m");
>>      conf.set("spark.eventLog.enabled", "true");
>>      conf.set("spark.eventLog.dir", "/tmp/");
>>      conf.set("spark.executor.extraClassPath", "/home/enlighted/ebd.jar");
>>      conf.set("spark.cores.max", "1");
>>      JavaSparkContext sc = new JavaSparkContext(conf);
>>
>>
>>      JavaRDD<String> cassandraRowsRDD = CassandraJavaUtil.javaFunctions(sc).cassandraTable("testing", "ec")
>>      .map(new Function<CassandraRow, String>() {
>>          private static final long serialVersionUID = -6263533266898869895L;
>>          @Override
>>          public String call(CassandraRow cassandraRow) throws Exception {
>>              return cassandraRow.toString();
>>          }
>>      });
>>      System.out.println("Data as CassandraRows: \n" + StringUtils.join(cassandraRowsRDD.toArray(), "\n"));
>>      sc.close();|
>>
>>
>> This job is stuck with insufficient resources warning. Here are logs:
>>
>>     1107 [main] INFO org.apache.spark.SparkContext  - Spark
>>     configuration:
>>     spark.app.name=Testing
>>     spark.cassandra.connection.host=192.168.4.129
>>     spark.cores.max=1
>>     spark.driver.maxResultSize=50m
>>     spark.eventLog.dir=/tmp/
>>     spark.eventLog.enabled=true
>>     spark.executor.extraClassPath=/home/enlighted/ebd.jar
>>     spark.executor.memory=200m
>>     spark.logConf=true
>>     spark.master=spark://192.168.4.134:7077
>>     1121 [main] INFO org.apache.spark.SecurityManager  - Changing
>>     view acls to: enlighted
>>     1122 [main] INFO org.apache.spark.SecurityManager  - Changing
>>     modify acls to: enlighted
>>     1123 [main] INFO org.apache.spark.SecurityManager  -
>>     SecurityManager: authentication disabled; ui acls disabled; users
>>     with view permissions: Set(enlighted); users with modify
>>     permissions: Set(enlighted)
>>     1767 [sparkDriver-akka.actor.default-dispatcher-4] INFO
>>     akka.event.slf4j.Slf4jLogger  - Slf4jLogger started
>>     1805 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting 
>>     - Starting remoting
>>     1957 [main] INFO org.apache.spark.util.Utils  - Successfully
>>     started service 'sparkDriver' on port 54611.
>>     1958 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting 
>>     - Remoting started; listening on addresses
>>     :[akka.tcp://sparkDriver@192.168.4.134:54611]
>>     1977 [main] INFO org.apache.spark.SparkEnv  - Registering
>>     MapOutputTracker
>>     1989 [main] INFO org.apache.spark.SparkEnv  - Registering
>>     BlockManagerMaster
>>     2007 [main] INFO org.apache.spark.storage.DiskBlockManager  -
>>     Created local directory at
>>     /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/blockmgr-3e3d54e7-16df-4e97-be48-b0c0fa0389e7
>>     2012 [main] INFO org.apache.spark.storage.MemoryStore  -
>>     MemoryStore started with capacity 456.0 MB
>>     2044 [main] INFO org.apache.spark.HttpFileServer  - HTTP File
>>     server directory is
>>     /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/httpd-64b4d92e-cde9-45fb-8b38-edc3cca3933c
>>     2046 [main] INFO org.apache.spark.HttpServer  - Starting HTTP Server
>>     2086 [main] INFO org.spark-project.jetty.server.Server  -
>>     jetty-8.y.z-SNAPSHOT
>>     2098 [main] INFO
>>     org.spark-project.jetty.server.AbstractConnector  - Started
>>     SocketConnector@0.0.0.0:44884
>>     2099 [main] INFO org.apache.spark.util.Utils  - Successfully
>>     started service 'HTTP file server' on port 44884.
>>     2108 [main] INFO org.apache.spark.SparkEnv  - Registering
>>     OutputCommitCoordinator
>>     2297 [main] INFO org.spark-project.jetty.server.Server  -
>>     jetty-8.y.z-SNAPSHOT
>>     2317 [main] INFO
>>     org.spark-project.jetty.server.AbstractConnector  - Started
>>     SelectChannelConnector@0.0.0.0:4040
>>     2318 [main] INFO org.apache.spark.util.Utils  - Successfully
>>     started service 'SparkUI' on port 4040.
>>     2320 [main] INFO org.apache.spark.ui.SparkUI  - Started SparkUI
>>     at http://192.168.4.134:4040
>>     2387 [sparkDriver-akka.actor.default-dispatcher-3] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  -
>>     Connecting to master
>>     akka.tcp://sparkMaster@192.168.4.134:7077/user/Master...
>>     2662 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     Connected to Spark cluster with app ID app-20150806054450-0001
>>     2680 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     added: app-20150806054450-0001/0 on
>>     worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566)
>>     with 1 cores
>>     2682 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     Granted executor ID app-20150806054450-0001/0 on hostPort
>>     192.168.4.129:45566 with 1 cores, 200.0 MB RAM
>>     2696 [main] INFO org.apache.spark.util.Utils  - Successfully
>>     started service
>>     'org.apache.spark.network.netty.NettyBlockTransferService' on
>>     port 49150.
>>     2696 [main] INFO
>>     org.apache.spark.network.netty.NettyBlockTransferService  -
>>     Server created on 49150
>>     2700 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     updated: app-20150806054450-0001/0 is now LOADING
>>     2706 [main] INFO org.apache.spark.storage.BlockManagerMaster -
>>     Trying to register BlockManager
>>     2708 [sparkDriver-akka.actor.default-dispatcher-17] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     updated: app-20150806054450-0001/0 is now RUNNING
>>     2710 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.storage.BlockManagerMasterEndpoint  -
>>     Registering block manager 192.168.4.134:49150 with 456.0 MB RAM,
>>     BlockManagerId(driver, 192.168.4.134, 49150)
>>     2713 [main] INFO org.apache.spark.storage.BlockManagerMaster -
>>     Registered BlockManager
>>     2922 [main] INFO org.apache.spark.scheduler.EventLoggingListener 
>>     - Logging events to file:/tmp/app-20150806054450-0001
>>     2939 [main] INFO
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     SchedulerBackend is ready for scheduling beginning after reached
>>     minRegisteredResourcesRatio: 0.0
>>     3321 [main] INFO com.datastax.driver.core.Cluster  - New
>>     Cassandra host /192.168.4.129:9042 added
>>     3321 [main] INFO com.datastax.driver.core.Cluster  - New
>>     Cassandra host /192.168.4.130:9042 added
>>     3322 [main] INFO
>>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
>>     Added host 192.168.4.130 (DC1)
>>     3322 [main] INFO com.datastax.driver.core.Cluster  - New
>>     Cassandra host /192.168.4.131:9042 added
>>     3323 [main] INFO
>>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
>>     Added host 192.168.4.131 (DC1)
>>     3323 [main] INFO com.datastax.driver.core.Cluster  - New
>>     Cassandra host /192.168.4.132:9042 added
>>     3323 [main] INFO
>>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy -
>>     Added host 192.168.4.132 (DC1)
>>     3325 [main] INFO
>>     com.datastax.spark.connector.cql.CassandraConnector  - Connected
>>     to Cassandra cluster: enldbcluster
>>     3881 [main] INFO org.apache.spark.SparkContext  - Starting job:
>>     toArray at Start.java:85
>>     3898 [pool-18-thread-1] INFO
>>     com.datastax.spark.connector.cql.CassandraConnector  -
>>     Disconnected from Cassandra cluster: enldbcluster
>>     3901 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Got job 0 (toArray at
>>     Start.java:85) with 6 output partitions (allowLocal=false)
>>     3902 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Final stage:
>>     ResultStage 0(toArray at Start.java:85)
>>     3902 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Parents of final
>>     stage: List()
>>     3908 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
>>     3925 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage
>>     0 (MapPartitionsRDD[1] at map at Start.java:77), which has no
>>     missing parents
>>     4002 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.storage.MemoryStore  - ensureFreeSpace(7488)
>>     called with curMem=0, maxMem=478182113
>>     4004 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.storage.MemoryStore  - Block broadcast_0 stored
>>     as values in memory (estimated size 7.3 KB, free 456.0 MB)
>>     4013 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.storage.MemoryStore  - ensureFreeSpace(4015)
>>     called with curMem=7488, maxMem=478182113
>>     4013 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.storage.MemoryStore  - Block broadcast_0_piece0
>>     stored as bytes in memory (estimated size 3.9 KB, free 456.0 MB)
>>     4015 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.storage.BlockManagerInfo  - Added
>>     broadcast_0_piece0 in memory on 192.168.4.134:49150 (size: 3.9
>>     KB, free: 456.0 MB)
>>     4017 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.SparkContext  - Created broadcast 0 from
>>     broadcast at DAGScheduler.scala:874
>>     4089 [dag-scheduler-event-loop] INFO
>>     com.datastax.driver.core.Cluster  - New Cassandra host
>>     /192.168.4.129:9042 added
>>     4089 [dag-scheduler-event-loop] INFO
>>     com.datastax.driver.core.Cluster  - New Cassandra host
>>     /192.168.4.130:9042 added
>>     4089 [dag-scheduler-event-loop] INFO
>>     com.datastax.driver.core.Cluster  - New Cassandra host
>>     /192.168.4.131:9042 added
>>     4089 [dag-scheduler-event-loop] INFO
>>     com.datastax.driver.core.Cluster  - New Cassandra host
>>     /192.168.4.132:9042 added
>>     4089 [dag-scheduler-event-loop] INFO
>>     com.datastax.spark.connector.cql.CassandraConnector  - Connected
>>     to Cassandra cluster: enldbcluster
>>     4394 [pool-18-thread-1] INFO
>>     com.datastax.spark.connector.cql.CassandraConnector  -
>>     Disconnected from Cassandra cluster: enldbcluster
>>     4806 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.DAGScheduler  - Submitting 6 missing
>>     tasks from ResultStage 0 (MapPartitionsRDD[1] at map at
>>     Start.java:77)
>>     4807 [dag-scheduler-event-loop] INFO
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set
>>     0.0 with 6 tasks
>>     19822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     34822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     49822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     64822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     79822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     94822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     109822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     124822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     124963 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     updated: app-20150806054450-0001/0 is now EXITED (Command exited
>>     with code 1)
>>     124964 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     Executor app-20150806054450-0001/0 removed: Command exited with
>>     code 1
>>     124968 [sparkDriver-akka.actor.default-dispatcher-17] ERROR
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     Asked to remove non-existent executor 0
>>     124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     added: app-20150806054450-0001/1 on
>>     worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566)
>>     with 1 cores
>>     124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>>     Granted executor ID app-20150806054450-0001/1 on hostPort
>>     192.168.4.129:45566 with 1 cores, 200.0 MB RAM
>>     124975 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     updated: app-20150806054450-0001/1 is now RUNNING
>>     125012 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>>     updated: app-20150806054450-0001/1 is now LOADING
>>     139822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     154822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     169823 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     184822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>     199822 [Timer-0] WARN
>>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>>     not accepted any resources; check your cluster UI to ensure that
>>     workers are registered and have sufficient resources
>>
>> ||
>> Please find attached the spark master UI and pom.xml file with 
>> dependencies.
>>
>> Can anyone please point out what could be an issue here.
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:user-unsubscribe@spark.apache.org
>> For additional commands, e-mail:user-help@spark.apache.org
>


Re: spark job not accepting resources from worker

Posted by Kushal Chokhani <ku...@enlightedinc.com>.
Any inputs?

In case of following message, is there a way to check which resources is 
not sufficient through some logs?

    [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl  -
    Initial job has not accepted any resources; check your cluster UI to
    ensure that workers are registered and have sufficient resources

Regards.

On 8/6/2015 11:40 AM, Kushal Chokhani wrote:
> Hi
>
> I have a spark/cassandra setup where I am using a spark cassandra java 
> connector to query on a table. So far, I have 1 spark master node (2 
> cores) and 1 worker node (4 cores). Both of them have following 
> spark-env.sh under conf/:
>
>     |#!/usr/bin/env bash
>     export SPARK_LOCAL_IP=127.0.0.1
>     export SPARK_MASTER_IP="192.168.4.134"
>     export SPARK_WORKER_MEMORY=1G
>     export SPARK_EXECUTOR_MEMORY=2G
>
>     |
>
> I am using spark 1.4.1 along with cassandra 2.2.0. I have started my 
> cassandra/spark setup. Created keyspace and table under cassandra and 
> added some rows on table. Now I try to run following spark job using 
> spark cassandra java connector:
>
> |     SparkConf conf = new SparkConf();
>      conf.setAppName("Testing");
>      conf.setMaster("spark://192.168.4.134:7077");
>      conf.set("spark.cassandra.connection.host", "192.168.4.129");
>      conf.set("spark.logConf", "true");
>      conf.set("spark.driver.maxResultSize", "50m");
>      conf.set("spark.executor.memory", "200m");
>      conf.set("spark.eventLog.enabled", "true");
>      conf.set("spark.eventLog.dir", "/tmp/");
>      conf.set("spark.executor.extraClassPath", "/home/enlighted/ebd.jar");
>      conf.set("spark.cores.max", "1");
>      JavaSparkContext sc = new JavaSparkContext(conf);
>
>
>      JavaRDD<String> cassandraRowsRDD = CassandraJavaUtil.javaFunctions(sc).cassandraTable("testing", "ec")
>      .map(new Function<CassandraRow, String>() {
>          private static final long serialVersionUID = -6263533266898869895L;
>          @Override
>          public String call(CassandraRow cassandraRow) throws Exception {
>              return cassandraRow.toString();
>          }
>      });
>      System.out.println("Data as CassandraRows: \n" + StringUtils.join(cassandraRowsRDD.toArray(), "\n"));
>      sc.close();|
>
>
> This job is stuck with insufficient resources warning. Here are logs:
>
>     1107 [main] INFO org.apache.spark.SparkContext  - Spark configuration:
>     spark.app.name=Testing
>     spark.cassandra.connection.host=192.168.4.129
>     spark.cores.max=1
>     spark.driver.maxResultSize=50m
>     spark.eventLog.dir=/tmp/
>     spark.eventLog.enabled=true
>     spark.executor.extraClassPath=/home/enlighted/ebd.jar
>     spark.executor.memory=200m
>     spark.logConf=true
>     spark.master=spark://192.168.4.134:7077
>     1121 [main] INFO org.apache.spark.SecurityManager  - Changing view
>     acls to: enlighted
>     1122 [main] INFO org.apache.spark.SecurityManager  - Changing
>     modify acls to: enlighted
>     1123 [main] INFO org.apache.spark.SecurityManager  -
>     SecurityManager: authentication disabled; ui acls disabled; users
>     with view permissions: Set(enlighted); users with modify
>     permissions: Set(enlighted)
>     1767 [sparkDriver-akka.actor.default-dispatcher-4] INFO
>     akka.event.slf4j.Slf4jLogger  - Slf4jLogger started
>     1805 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting 
>     - Starting remoting
>     1957 [main] INFO org.apache.spark.util.Utils  - Successfully
>     started service 'sparkDriver' on port 54611.
>     1958 [sparkDriver-akka.actor.default-dispatcher-4] INFO Remoting 
>     - Remoting started; listening on addresses
>     :[akka.tcp://sparkDriver@192.168.4.134:54611]
>     1977 [main] INFO org.apache.spark.SparkEnv  - Registering
>     MapOutputTracker
>     1989 [main] INFO org.apache.spark.SparkEnv  - Registering
>     BlockManagerMaster
>     2007 [main] INFO org.apache.spark.storage.DiskBlockManager  -
>     Created local directory at
>     /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/blockmgr-3e3d54e7-16df-4e97-be48-b0c0fa0389e7
>     2012 [main] INFO org.apache.spark.storage.MemoryStore  -
>     MemoryStore started with capacity 456.0 MB
>     2044 [main] INFO org.apache.spark.HttpFileServer  - HTTP File
>     server directory is
>     /tmp/spark-f21125fd-ae9d-460e-884d-563fa8720f09/httpd-64b4d92e-cde9-45fb-8b38-edc3cca3933c
>     2046 [main] INFO org.apache.spark.HttpServer  - Starting HTTP Server
>     2086 [main] INFO org.spark-project.jetty.server.Server  -
>     jetty-8.y.z-SNAPSHOT
>     2098 [main] INFO org.spark-project.jetty.server.AbstractConnector 
>     - Started SocketConnector@0.0.0.0:44884
>     2099 [main] INFO org.apache.spark.util.Utils  - Successfully
>     started service 'HTTP file server' on port 44884.
>     2108 [main] INFO org.apache.spark.SparkEnv  - Registering
>     OutputCommitCoordinator
>     2297 [main] INFO org.spark-project.jetty.server.Server  -
>     jetty-8.y.z-SNAPSHOT
>     2317 [main] INFO org.spark-project.jetty.server.AbstractConnector 
>     - Started SelectChannelConnector@0.0.0.0:4040
>     2318 [main] INFO org.apache.spark.util.Utils  - Successfully
>     started service 'SparkUI' on port 4040.
>     2320 [main] INFO org.apache.spark.ui.SparkUI  - Started SparkUI at
>     http://192.168.4.134:4040
>     2387 [sparkDriver-akka.actor.default-dispatcher-3] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Connecting
>     to master akka.tcp://sparkMaster@192.168.4.134:7077/user/Master...
>     2662 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     Connected to Spark cluster with app ID app-20150806054450-0001
>     2680 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     added: app-20150806054450-0001/0 on
>     worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566)
>     with 1 cores
>     2682 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     Granted executor ID app-20150806054450-0001/0 on hostPort
>     192.168.4.129:45566 with 1 cores, 200.0 MB RAM
>     2696 [main] INFO org.apache.spark.util.Utils  - Successfully
>     started service
>     'org.apache.spark.network.netty.NettyBlockTransferService' on port
>     49150.
>     2696 [main] INFO
>     org.apache.spark.network.netty.NettyBlockTransferService  - Server
>     created on 49150
>     2700 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     updated: app-20150806054450-0001/0 is now LOADING
>     2706 [main] INFO org.apache.spark.storage.BlockManagerMaster  -
>     Trying to register BlockManager
>     2708 [sparkDriver-akka.actor.default-dispatcher-17] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     updated: app-20150806054450-0001/0 is now RUNNING
>     2710 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.storage.BlockManagerMasterEndpoint  - Registering
>     block manager 192.168.4.134:49150 with 456.0 MB RAM,
>     BlockManagerId(driver, 192.168.4.134, 49150)
>     2713 [main] INFO org.apache.spark.storage.BlockManagerMaster  -
>     Registered BlockManager
>     2922 [main] INFO org.apache.spark.scheduler.EventLoggingListener 
>     - Logging events to file:/tmp/app-20150806054450-0001
>     2939 [main] INFO
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     SchedulerBackend is ready for scheduling beginning after reached
>     minRegisteredResourcesRatio: 0.0
>     3321 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
>     host /192.168.4.129:9042 added
>     3321 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
>     host /192.168.4.130:9042 added
>     3322 [main] INFO
>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy
>     - Added host 192.168.4.130 (DC1)
>     3322 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
>     host /192.168.4.131:9042 added
>     3323 [main] INFO
>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy
>     - Added host 192.168.4.131 (DC1)
>     3323 [main] INFO com.datastax.driver.core.Cluster  - New Cassandra
>     host /192.168.4.132:9042 added
>     3323 [main] INFO
>     com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy
>     - Added host 192.168.4.132 (DC1)
>     3325 [main] INFO
>     com.datastax.spark.connector.cql.CassandraConnector  - Connected
>     to Cassandra cluster: enldbcluster
>     3881 [main] INFO org.apache.spark.SparkContext  - Starting job:
>     toArray at Start.java:85
>     3898 [pool-18-thread-1] INFO
>     com.datastax.spark.connector.cql.CassandraConnector  -
>     Disconnected from Cassandra cluster: enldbcluster
>     3901 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Got job 0 (toArray at
>     Start.java:85) with 6 output partitions (allowLocal=false)
>     3902 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Final stage:
>     ResultStage 0(toArray at Start.java:85)
>     3902 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Parents of final stage:
>     List()
>     3908 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
>     3925 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage
>     0 (MapPartitionsRDD[1] at map at Start.java:77), which has no
>     missing parents
>     4002 [dag-scheduler-event-loop] INFO
>     org.apache.spark.storage.MemoryStore  - ensureFreeSpace(7488)
>     called with curMem=0, maxMem=478182113
>     4004 [dag-scheduler-event-loop] INFO
>     org.apache.spark.storage.MemoryStore  - Block broadcast_0 stored
>     as values in memory (estimated size 7.3 KB, free 456.0 MB)
>     4013 [dag-scheduler-event-loop] INFO
>     org.apache.spark.storage.MemoryStore  - ensureFreeSpace(4015)
>     called with curMem=7488, maxMem=478182113
>     4013 [dag-scheduler-event-loop] INFO
>     org.apache.spark.storage.MemoryStore  - Block broadcast_0_piece0
>     stored as bytes in memory (estimated size 3.9 KB, free 456.0 MB)
>     4015 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.storage.BlockManagerInfo  - Added
>     broadcast_0_piece0 in memory on 192.168.4.134:49150 (size: 3.9 KB,
>     free: 456.0 MB)
>     4017 [dag-scheduler-event-loop] INFO
>     org.apache.spark.SparkContext  - Created broadcast 0 from
>     broadcast at DAGScheduler.scala:874
>     4089 [dag-scheduler-event-loop] INFO
>     com.datastax.driver.core.Cluster  - New Cassandra host
>     /192.168.4.129:9042 added
>     4089 [dag-scheduler-event-loop] INFO
>     com.datastax.driver.core.Cluster  - New Cassandra host
>     /192.168.4.130:9042 added
>     4089 [dag-scheduler-event-loop] INFO
>     com.datastax.driver.core.Cluster  - New Cassandra host
>     /192.168.4.131:9042 added
>     4089 [dag-scheduler-event-loop] INFO
>     com.datastax.driver.core.Cluster  - New Cassandra host
>     /192.168.4.132:9042 added
>     4089 [dag-scheduler-event-loop] INFO
>     com.datastax.spark.connector.cql.CassandraConnector  - Connected
>     to Cassandra cluster: enldbcluster
>     4394 [pool-18-thread-1] INFO
>     com.datastax.spark.connector.cql.CassandraConnector  -
>     Disconnected from Cassandra cluster: enldbcluster
>     4806 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.DAGScheduler  - Submitting 6 missing
>     tasks from ResultStage 0 (MapPartitionsRDD[1] at map at Start.java:77)
>     4807 [dag-scheduler-event-loop] INFO
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set
>     0.0 with 6 tasks
>     19822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     34822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     49822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     64822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     79822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     94822 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl 
>     - Initial job has not accepted any resources; check your cluster
>     UI to ensure that workers are registered and have sufficient resources
>     109822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     124822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     124963 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     updated: app-20150806054450-0001/0 is now EXITED (Command exited
>     with code 1)
>     124964 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     Executor app-20150806054450-0001/0 removed: Command exited with code 1
>     124968 [sparkDriver-akka.actor.default-dispatcher-17] ERROR
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     Asked to remove non-existent executor 0
>     124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     added: app-20150806054450-0001/1 on
>     worker-20150806053100-192.168.4.129-45566 (192.168.4.129:45566)
>     with 1 cores
>     124969 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend -
>     Granted executor ID app-20150806054450-0001/1 on hostPort
>     192.168.4.129:45566 with 1 cores, 200.0 MB RAM
>     124975 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     updated: app-20150806054450-0001/1 is now RUNNING
>     125012 [sparkDriver-akka.actor.default-dispatcher-14] INFO
>     org.apache.spark.deploy.client.AppClient$ClientActor  - Executor
>     updated: app-20150806054450-0001/1 is now LOADING
>     139822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     154822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     169823 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     184822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>     199822 [Timer-0] WARN
>     org.apache.spark.scheduler.TaskSchedulerImpl  - Initial job has
>     not accepted any resources; check your cluster UI to ensure that
>     workers are registered and have sufficient resources
>
> ||
> Please find attached the spark master UI and pom.xml file with 
> dependencies.
>
> Can anyone please point out what could be an issue here.
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org