You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eran Witkon <er...@gmail.com> on 2015/12/17 21:25:39 UTC
Can't run spark on yarn
Hi,
I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
spark-env.sh
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
bash_profile
#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_USER_CLASSPATH_FIRST=true
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
#HADOOP VARIABLES END
export SPARK_HOME=/usr/local/spark
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
When I run spark-shell
./bin/spark-shell --master yarn-client
Output:
15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:08 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:38389
15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP class
server' on port 38389.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.2
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_66)
Type in expressions to have them evaluated.
Type :help for more information.
15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
interface wlp1s0)
15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
to another address
15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/12/17 22:22:11 INFO Remoting: Starting remoting
15/12/17 22:22:12 INFO util.Utils: Successfully started service
'sparkDriver' on port 36381.
15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@10.0.0.1:36381]
15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
capacity 530.0 MB
15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:12 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:36760
15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file
server' on port 36760.
15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/12/17 22:22:12 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
on port 4040.
15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040
15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
DAGScheduler for source because spark.app.id is not set.
15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
0.0.0.0:8032
15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
cluster with 1 NodeManagers
15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not
requested more than the maximum memory capability of the cluster (8192 MB
per container)
15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context for
our AM
15/12/17 22:22:12 INFO yarn.Client: Setting up the launch environment for
our AM container
15/12/17 22:22:12 INFO yarn.Client: Preparing resources for our AM container
15/12/17 22:22:13 INFO yarn.Client: Uploading resource
file:/usr/local/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar ->
hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/spark-assembly-1.5.2-hadoop2.6.0.jar
15/12/17 22:22:15 INFO yarn.Client: Uploading resource
file:/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/__spark_conf__3605678380745697769.zip
->
hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/__spark_conf__3605678380745697769.zip
15/12/17 22:22:15 INFO spark.SecurityManager: Changing view acls to: hduser
15/12/17 22:22:15 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/12/17 22:22:15 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/12/17 22:22:15 INFO yarn.Client: Submitting application 3 to
ResourceManager
15/12/17 22:22:15 INFO impl.YarnClientImpl: Submitted application
application_1450383657924_0003
15/12/17 22:22:16 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: ACCEPTED)
15/12/17 22:22:16 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1450383735512
final status: UNDEFINED
tracking URL:
http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
user: hduser
15/12/17 22:22:17 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: ACCEPTED)
15/12/17 22:22:18 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: ACCEPTED)
15/12/17 22:22:19 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: ACCEPTED)
15/12/17 22:22:20 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: ACCEPTED)
15/12/17 22:22:21 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://
sparkYarnAM@10.0.0.1:34719/user/YarnAM#-313490512])
15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI
Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
/proxy/application_1450383657924_0003
15/12/17 22:22:21 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/12/17 22:22:21 INFO yarn.Client: Application report for
application_1450383657924_0003 (state: RUNNING)
15/12/17 22:22:21 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.0.0.1
ApplicationMaster RPC port: 0
queue: default
start time: 1450383735512
final status: UNDEFINED
tracking URL:
http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
user: hduser
15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Application
application_1450383657924_0003 has started running.
15/12/17 22:22:21 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 33155.
15/12/17 22:22:21 INFO netty.NettyBlockTransferService: Server created on
33155
15/12/17 22:22:21 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/12/17 22:22:21 INFO storage.BlockManagerMasterEndpoint: Registering
block manager 10.0.0.1:33155 with 530.0 MB RAM, BlockManagerId(driver,
10.0.0.1, 33155)
15/12/17 22:22:21 INFO storage.BlockManagerMaster: Registered BlockManager
15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster has disassociated: 10.0.0.1:34719
15/12/17 22:22:22 WARN remote.ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkYarnAM@10.0.0.1:34719] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
15/12/17 22:22:22 INFO scheduler.EventLoggingListener: Logging events to
hdfs://localhost:54310/spark-log/application_1450383657924_0003
15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster has disassociated: 10.0.0.1:34719
15/12/17 22:22:25 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://
sparkYarnAM@10.0.0.1:39353/user/YarnAM#1861872417])
15/12/17 22:22:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI
Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
/proxy/application_1450383657924_0003
15/12/17 22:22:25 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster has disassociated: 10.0.0.1:39353
15/12/17 22:22:30 WARN remote.ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkYarnAM@10.0.0.1:39353] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster has disassociated: 10.0.0.1:39353
15/12/17 22:22:30 ERROR cluster.YarnClientSchedulerBackend: Yarn
application has already exited with state FINISHED!
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/api,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
15/12/17 22:22:30 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
15/12/17 22:22:31 INFO ui.SparkUI: Stopped Spark web UI at
http://10.0.0.1:4040
15/12/17 22:22:31 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Shutting down
all executors
15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Asking each
executor to shut down
15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Stopped
15/12/17 22:22:36 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
15/12/17 22:22:36 INFO storage.MemoryStore: MemoryStore cleared
15/12/17 22:22:36 INFO storage.BlockManager: BlockManager stopped
15/12/17 22:22:36 INFO storage.BlockManagerMaster: BlockManagerMaster
stopped
15/12/17 22:22:36 INFO spark.SparkContext: Successfully stopped SparkContext
15/12/17 22:22:36 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Shutting down remote daemon.
15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Remote daemon shut down; proceeding with flushing remote transports.
15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Remoting shut down.
15/12/17 22:22:42 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend
is ready for scheduling beginning after waiting
maxRegisteredResourcesWaitingTime: 30000(ms)
15/12/17 22:22:42 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
15/12/17 22:22:49 INFO hive.HiveContext: Initializing execution hive,
version 1.2.1
15/12/17 22:22:49 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0
15/12/17 22:22:49 INFO client.ClientWrapper: Loaded
org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
15/12/17 22:22:49 INFO metastore.HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/12/17 22:22:49 INFO metastore.ObjectStore: ObjectStore, initialize called
15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
datanucleus.cache.level2 unknown - will be ignored
15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
present in CLASSPATH (or one of dependencies)
15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
present in CLASSPATH (or one of dependencies)
15/12/17 22:22:59 INFO metastore.ObjectStore: Setting MetaStore object pin
classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:02 INFO metastore.MetaStoreDirectSql: Using direct SQL,
underlying DB is DERBY
15/12/17 22:23:02 INFO metastore.ObjectStore: Initialized ObjectStore
15/12/17 22:23:02 WARN metastore.ObjectStore: Version information not found
in metastore. hive.metastore.schema.verification is not enabled so
recording the schema version 1.2.0
15/12/17 22:23:02 WARN metastore.ObjectStore: Failed to get database
default, returning NoSuchObjectException
15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added admin role in
metastore
15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added public role in
metastore
15/12/17 22:23:03 INFO metastore.HiveMetaStore: No user is added in admin
role, since config is empty
15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_all_databases
15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
cmd=get_all_databases
15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_functions:
db=default pat=*
15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser
ip=unknown-ip-addr cmd=get_functions:
db=default pat=*
15/12/17 22:23:03 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:09 INFO session.SessionState: Created local directory:
/tmp/eae419f8-7dad-417b-bc3e-77c048c88337_resources
15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
/tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
15/12/17 22:23:09 INFO session.SessionState: Created local directory:
/tmp/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
/tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337/_tmp_space.db
15/12/17 22:23:09 INFO hive.HiveContext: default warehouse location is
/user/hive/warehouse
15/12/17 22:23:09 INFO hive.HiveContext: Initializing
HiveMetastoreConnection version 1.2.1 using Spark classes.
15/12/17 22:23:09 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0
15/12/17 22:23:09 INFO client.ClientWrapper: Loaded
org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
15/12/17 22:23:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/12/17 22:23:10 INFO metastore.HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/12/17 22:23:10 INFO metastore.ObjectStore: ObjectStore, initialize called
15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
datanucleus.cache.level2 unknown - will be ignored
15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
present in CLASSPATH (or one of dependencies)
15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
present in CLASSPATH (or one of dependencies)
15/12/17 22:23:11 INFO metastore.ObjectStore: Setting MetaStore object pin
classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:12 INFO DataNucleus.Query: Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
closing
15/12/17 22:23:12 INFO metastore.MetaStoreDirectSql: Using direct SQL,
underlying DB is DERBY
15/12/17 22:23:12 INFO metastore.ObjectStore: Initialized ObjectStore
15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added admin role in
metastore
15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added public role in
metastore
15/12/17 22:23:13 INFO metastore.HiveMetaStore: No user is added in admin
role, since config is empty
15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_all_databases
15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
cmd=get_all_databases
15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_functions:
db=default pat=*
15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser
ip=unknown-ip-addr cmd=get_functions:
db=default pat=*
15/12/17 22:23:13 INFO DataNucleus.Datastore: The class
"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
"embedded-only" so does not have its own datastore table.
15/12/17 22:23:18 INFO session.SessionState: Created local directory:
/tmp/7f892b9d-38ef-4fab-bc44-c1afaf07f50f_resources
15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
/tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
15/12/17 22:23:18 INFO session.SessionState: Created local directory:
/tmp/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
/tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f/_tmp_space.db
15/12/17 22:23:18 INFO repl.SparkILoop: Created sql context (with Hive
support)..
SQL context available as sqlContext.
scala>
When I am running the following code:
val data = sc.wholeTextFiles("hdfs://localhost:54310/spark-log/*")
I am getting exception:
java.lang.IllegalStateException: Cannot call methods on a stopped
SparkContext
at org.apache.spark.SparkContext.org
$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076)
at
org.apache.spark.SparkContext.wholeTextFiles$default$2(SparkContext.scala:864)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:21)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:32)
at $iwC$$iwC$$iwC.<init>(<console>:34)
at $iwC$$iwC.<init>(<console>:36)
at $iwC.<init>(<console>:38)
at <init>(<console>:40)
at .<init>(<console>:44)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
*What is wrong here? how should I configure spark to use yarn and hive
correctly?*
Eran
Re: Can't run spark on yarn
Posted by Alexander Pivovarov <ap...@gmail.com>.
Try to start aws EMR 4.2.0 with hadoop and spark applications on spot
instances. Then look at how hadoop and spark configured. Try to configure
your hadoop and spark similar way
On Dec 17, 2015 6:09 PM, "Saisai Shao" <sa...@gmail.com> wrote:
> Please check the Yarn AM log to see why AM is failed to start. That's the
> reason why using `sc` will get such complaint.
>
> On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon <er...@gmail.com> wrote:
>
>> Hi,
>> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>>
>> spark-env.sh
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>>
>> bash_profile
>> #HADOOP VARIABLES START
>> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>> export HADOOP_INSTALL=/usr/local/hadoop
>> export PATH=$PATH:$HADOOP_INSTALL/bin
>> export PATH=$PATH:$HADOOP_INSTALL/sbin
>> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
>> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
>> export YARN_HOME=$HADOOP_INSTALL
>> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
>> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
>> export HADOOP_USER_CLASSPATH_FIRST=true
>> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
>> #HADOOP VARIABLES END
>>
>> export SPARK_HOME=/usr/local/spark
>> export HIVE_HOME=/usr/local/hive
>> export PATH=$PATH:$HIVE_HOME/bin
>>
>>
>> When I run spark-shell
>> ./bin/spark-shell --master yarn-client
>>
>> Output:
>> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:38389
>> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
>> class server' on port 38389.
>> Welcome to
>> ____ __
>> / __/__ ___ _____/ /__
>> _\ \/ _ \/ _ `/ __/ '_/
>> /___/ .__/\_,_/_/ /_/\_\ version 1.5.2
>> /_/
>>
>> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
>> 1.8.0_66)
>> Type in expressions to have them evaluated.
>> Type :help for more information.
>> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
>> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
>> interface wlp1s0)
>> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
>> to another address
>> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> 15/12/17 22:22:11 INFO Remoting: Starting remoting
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
>> 'sparkDriver' on port 36381.
>> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkDriver@10.0.0.1:36381]
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
>> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
>> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
>> capacity 530.0 MB
>> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory
>> is
>> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
>> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SocketConnector@0.0.0.0:36760
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP
>> file server' on port 36760.
>> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:4040
>> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
>> on port 4040.
>> 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at
>> http://10.0.0.1:4040
>> 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
>> DAGScheduler for source because spark.app.id is not set.
>> 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
>> 0.0.0.0:8032
>> 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
>> cluster with 1 NodeManagers
>> 15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not
>> requested more than the maximum memory capability of the cluster (8192 MB
>> per container)
>> 15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896
>> MB memory including 384 MB overhead
>> 15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context
>> for our AM
>> 15/12/17 22:22:12 INFO yarn.Client: Setting up the launch environment for
>> our AM container
>> 15/12/17 22:22:12 INFO yarn.Client: Preparing resources for our AM
>> container
>> 15/12/17 22:22:13 INFO yarn.Client: Uploading resource
>> file:/usr/local/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar ->
>> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/spark-assembly-1.5.2-hadoop2.6.0.jar
>> 15/12/17 22:22:15 INFO yarn.Client: Uploading resource
>> file:/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/__spark_conf__3605678380745697769.zip
>> ->
>> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/__spark_conf__3605678380745697769.zip
>> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing view acls to:
>> hduser
>> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing modify acls to:
>> hduser
>> 15/12/17 22:22:15 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hduser); users with modify permissions: Set(hduser)
>> 15/12/17 22:22:15 INFO yarn.Client: Submitting application 3 to
>> ResourceManager
>> 15/12/17 22:22:15 INFO impl.YarnClientImpl: Submitted application
>> application_1450383657924_0003
>> 15/12/17 22:22:16 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: ACCEPTED)
>> 15/12/17 22:22:16 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: default
>> start time: 1450383735512
>> final status: UNDEFINED
>> tracking URL:
>> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
>> user: hduser
>> 15/12/17 22:22:17 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: ACCEPTED)
>> 15/12/17 22:22:18 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: ACCEPTED)
>> 15/12/17 22:22:19 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: ACCEPTED)
>> 15/12/17 22:22:20 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: ACCEPTED)
>> 15/12/17 22:22:21 INFO
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster
>> registered as AkkaRpcEndpointRef(Actor[akka.tcp://
>> sparkYarnAM@10.0.0.1:34719/user/YarnAM#-313490512])
>> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI
>> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
>> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
>> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
>> /proxy/application_1450383657924_0003
>> 15/12/17 22:22:21 INFO ui.JettyUtils: Adding filter:
>> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>> 15/12/17 22:22:21 INFO yarn.Client: Application report for
>> application_1450383657924_0003 (state: RUNNING)
>> 15/12/17 22:22:21 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: 10.0.0.1
>> ApplicationMaster RPC port: 0
>> queue: default
>> start time: 1450383735512
>> final status: UNDEFINED
>> tracking URL:
>> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
>> user: hduser
>> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Application
>> application_1450383657924_0003 has started running.
>> 15/12/17 22:22:21 INFO util.Utils: Successfully started service
>> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33155.
>> 15/12/17 22:22:21 INFO netty.NettyBlockTransferService: Server created on
>> 33155
>> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Trying to register
>> BlockManager
>> 15/12/17 22:22:21 INFO storage.BlockManagerMasterEndpoint: Registering
>> block manager 10.0.0.1:33155 with 530.0 MB RAM, BlockManagerId(driver,
>> 10.0.0.1, 33155)
>> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Registered BlockManager
>> 15/12/17 22:22:22 WARN
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has
>> disassociated: 10.0.0.1:34719
>> 15/12/17 22:22:22 WARN remote.ReliableDeliverySupervisor: Association
>> with remote system [akka.tcp://sparkYarnAM@10.0.0.1:34719] has failed,
>> address is now gated for [5000] ms. Reason: [Disassociated]
>> 15/12/17 22:22:22 INFO scheduler.EventLoggingListener: Logging events to
>> hdfs://localhost:54310/spark-log/application_1450383657924_0003
>> 15/12/17 22:22:22 WARN
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has
>> disassociated: 10.0.0.1:34719
>> 15/12/17 22:22:25 INFO
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster
>> registered as AkkaRpcEndpointRef(Actor[akka.tcp://
>> sparkYarnAM@10.0.0.1:39353/user/YarnAM#1861872417])
>> 15/12/17 22:22:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI
>> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
>> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
>> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
>> /proxy/application_1450383657924_0003
>> 15/12/17 22:22:25 INFO ui.JettyUtils: Adding filter:
>> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>> 15/12/17 22:22:30 WARN
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has
>> disassociated: 10.0.0.1:39353
>> 15/12/17 22:22:30 WARN remote.ReliableDeliverySupervisor: Association
>> with remote system [akka.tcp://sparkYarnAM@10.0.0.1:39353] has failed,
>> address is now gated for [5000] ms. Reason: [Disassociated]
>> 15/12/17 22:22:30 WARN
>> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has
>> disassociated: 10.0.0.1:39353
>> 15/12/17 22:22:30 ERROR cluster.YarnClientSchedulerBackend: Yarn
>> application has already exited with state FINISHED!
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/metrics/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/api,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/static,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/executors/threadDump,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/executors/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/executors,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/environment/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/environment,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/storage/rdd,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/storage/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/storage,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/pool/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/pool,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/stage/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/stage,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/stages,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/jobs/job/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/jobs/job,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/jobs/json,null}
>> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
>> o.s.j.s.ServletContextHandler{/jobs,null}
>> 15/12/17 22:22:31 INFO ui.SparkUI: Stopped Spark web UI at
>> http://10.0.0.1:4040
>> 15/12/17 22:22:31 INFO scheduler.DAGScheduler: Stopping DAGScheduler
>> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Shutting down
>> all executors
>> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Asking each
>> executor to shut down
>> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Stopped
>> 15/12/17 22:22:36 INFO spark.MapOutputTrackerMasterEndpoint:
>> MapOutputTrackerMasterEndpoint stopped!
>> 15/12/17 22:22:36 INFO storage.MemoryStore: MemoryStore cleared
>> 15/12/17 22:22:36 INFO storage.BlockManager: BlockManager stopped
>> 15/12/17 22:22:36 INFO storage.BlockManagerMaster: BlockManagerMaster
>> stopped
>> 15/12/17 22:22:36 INFO spark.SparkContext: Successfully stopped
>> SparkContext
>> 15/12/17 22:22:36 INFO
>> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
>> OutputCommitCoordinator stopped!
>> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
>> Shutting down remote daemon.
>> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
>> Remote daemon shut down; proceeding with flushing remote transports.
>> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
>> Remoting shut down.
>> 15/12/17 22:22:42 INFO cluster.YarnClientSchedulerBackend:
>> SchedulerBackend is ready for scheduling beginning after waiting
>> maxRegisteredResourcesWaitingTime: 30000(ms)
>> 15/12/17 22:22:42 INFO repl.SparkILoop: Created spark context..
>> Spark context available as sc.
>> 15/12/17 22:22:49 INFO hive.HiveContext: Initializing execution hive,
>> version 1.2.1
>> 15/12/17 22:22:49 INFO client.ClientWrapper: Inspected Hadoop version:
>> 2.6.0
>> 15/12/17 22:22:49 INFO client.ClientWrapper: Loaded
>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
>> 15/12/17 22:22:49 INFO metastore.HiveMetaStore: 0: Opening raw store with
>> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>> 15/12/17 22:22:49 INFO metastore.ObjectStore: ObjectStore, initialize
>> called
>> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
>> datanucleus.cache.level2 unknown - will be ignored
>> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
>> present in CLASSPATH (or one of dependencies)
>> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
>> present in CLASSPATH (or one of dependencies)
>> 15/12/17 22:22:59 INFO metastore.ObjectStore: Setting MetaStore object
>> pin classes with
>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:02 INFO metastore.MetaStoreDirectSql: Using direct SQL,
>> underlying DB is DERBY
>> 15/12/17 22:23:02 INFO metastore.ObjectStore: Initialized ObjectStore
>> 15/12/17 22:23:02 WARN metastore.ObjectStore: Version information not
>> found in metastore. hive.metastore.schema.verification is not enabled so
>> recording the schema version 1.2.0
>> 15/12/17 22:23:02 WARN metastore.ObjectStore: Failed to get database
>> default, returning NoSuchObjectException
>> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added admin role in
>> metastore
>> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added public role in
>> metastore
>> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: No user is added in admin
>> role, since config is empty
>> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_all_databases
>> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
>> cmd=get_all_databases
>> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_functions:
>> db=default pat=*
>> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_functions:
>> db=default pat=*
>> 15/12/17 22:23:03 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:09 INFO session.SessionState: Created local directory:
>> /tmp/eae419f8-7dad-417b-bc3e-77c048c88337_resources
>> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
>> 15/12/17 22:23:09 INFO session.SessionState: Created local directory:
>> /tmp/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
>> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337/_tmp_space.db
>> 15/12/17 22:23:09 INFO hive.HiveContext: default warehouse location is
>> /user/hive/warehouse
>> 15/12/17 22:23:09 INFO hive.HiveContext: Initializing
>> HiveMetastoreConnection version 1.2.1 using Spark classes.
>> 15/12/17 22:23:09 INFO client.ClientWrapper: Inspected Hadoop version:
>> 2.6.0
>> 15/12/17 22:23:09 INFO client.ClientWrapper: Loaded
>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
>> 15/12/17 22:23:10 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 15/12/17 22:23:10 INFO metastore.HiveMetaStore: 0: Opening raw store with
>> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>> 15/12/17 22:23:10 INFO metastore.ObjectStore: ObjectStore, initialize
>> called
>> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
>> hive.metastore.integral.jdo.pushdown unknown - will be ignored
>> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
>> datanucleus.cache.level2 unknown - will be ignored
>> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
>> present in CLASSPATH (or one of dependencies)
>> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
>> present in CLASSPATH (or one of dependencies)
>> 15/12/17 22:23:11 INFO metastore.ObjectStore: Setting MetaStore object
>> pin classes with
>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
>> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:12 INFO DataNucleus.Query: Reading in results for query
>> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used
>> is closing
>> 15/12/17 22:23:12 INFO metastore.MetaStoreDirectSql: Using direct SQL,
>> underlying DB is DERBY
>> 15/12/17 22:23:12 INFO metastore.ObjectStore: Initialized ObjectStore
>> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added admin role in
>> metastore
>> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added public role in
>> metastore
>> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: No user is added in admin
>> role, since config is empty
>> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_all_databases
>> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
>> cmd=get_all_databases
>> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_functions:
>> db=default pat=*
>> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_functions:
>> db=default pat=*
>> 15/12/17 22:23:13 INFO DataNucleus.Datastore: The class
>> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
>> "embedded-only" so does not have its own datastore table.
>> 15/12/17 22:23:18 INFO session.SessionState: Created local directory:
>> /tmp/7f892b9d-38ef-4fab-bc44-c1afaf07f50f_resources
>> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
>> 15/12/17 22:23:18 INFO session.SessionState: Created local directory:
>> /tmp/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
>> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
>> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f/_tmp_space.db
>> 15/12/17 22:23:18 INFO repl.SparkILoop: Created sql context (with Hive
>> support)..
>> SQL context available as sqlContext.
>>
>> scala>
>>
>> When I am running the following code:
>> val data = sc.wholeTextFiles("hdfs://localhost:54310/spark-log/*")
>>
>> I am getting exception:
>> java.lang.IllegalStateException: Cannot call methods on a stopped
>> SparkContext
>> at org.apache.spark.SparkContext.org
>> $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
>> at
>> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063)
>> at
>> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076)
>> at
>> org.apache.spark.SparkContext.wholeTextFiles$default$2(SparkContext.scala:864)
>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:21)
>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
>> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
>> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:32)
>> at $iwC$$iwC$$iwC.<init>(<console>:34)
>> at $iwC$$iwC.<init>(<console>:36)
>> at $iwC.<init>(<console>:38)
>> at <init>(<console>:40)
>> at .<init>(<console>:44)
>> at .<clinit>(<console>)
>> at .<init>(<console>:7)
>> at .<clinit>(<console>)
>> at $print(<console>)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> at
>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>> at
>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
>> at
>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
>> at org.apache.spark.repl.SparkILoop.org
>> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>> at
>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>> at
>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.org
>> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>> at org.apache.spark.repl.Main.main(Main.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> *What is wrong here? how should I configure spark to use yarn and hive
>> correctly?*
>>
>> Eran
>>
>>
>
Re: Can't run spark on yarn
Posted by Saisai Shao <sa...@gmail.com>.
Please check the Yarn AM log to see why AM is failed to start. That's the
reason why using `sc` will get such complaint.
On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon <er...@gmail.com> wrote:
> Hi,
> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn
>
> spark-env.sh
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
>
> bash_profile
> #HADOOP VARIABLES START
> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
> export HADOOP_INSTALL=/usr/local/hadoop
> export PATH=$PATH:$HADOOP_INSTALL/bin
> export PATH=$PATH:$HADOOP_INSTALL/sbin
> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_HOME=$HADOOP_INSTALL
> export HADOOP_HDFS_HOME=$HADOOP_INSTALL
> export YARN_HOME=$HADOOP_INSTALL
> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
> export HADOOP_USER_CLASSPATH_FIRST=true
> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
> #HADOOP VARIABLES END
>
> export SPARK_HOME=/usr/local/spark
> export HIVE_HOME=/usr/local/hive
> export PATH=$PATH:$HIVE_HOME/bin
>
>
> When I run spark-shell
> ./bin/spark-shell --master yarn-client
>
> Output:
> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:08 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:38389
> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP
> class server' on port 38389.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 1.5.2
> /_/
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_66)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro
> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on
> interface wlp1s0)
> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind
> to another address
> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/12/17 22:22:11 INFO Remoting: Starting remoting
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service
> 'sparkDriver' on port 36381.
> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@10.0.0.1:36381]
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory
> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d
> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with
> capacity 530.0 MB
> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00
> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:36760
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 36760.
> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/12/17 22:22:12 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI'
> on port 4040.
> 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at http://10.0.0.1:4040
> 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name
> DAGScheduler for source because spark.app.id is not set.
> 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from
> cluster with 1 NodeManagers
> 15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not
> requested more than the maximum memory capability of the cluster (8192 MB
> per container)
> 15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896
> MB memory including 384 MB overhead
> 15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context
> for our AM
> 15/12/17 22:22:12 INFO yarn.Client: Setting up the launch environment for
> our AM container
> 15/12/17 22:22:12 INFO yarn.Client: Preparing resources for our AM
> container
> 15/12/17 22:22:13 INFO yarn.Client: Uploading resource
> file:/usr/local/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar ->
> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/spark-assembly-1.5.2-hadoop2.6.0.jar
> 15/12/17 22:22:15 INFO yarn.Client: Uploading resource
> file:/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/__spark_conf__3605678380745697769.zip
> ->
> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/__spark_conf__3605678380745697769.zip
> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing view acls to: hduser
> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing modify acls to:
> hduser
> 15/12/17 22:22:15 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hduser); users with modify permissions: Set(hduser)
> 15/12/17 22:22:15 INFO yarn.Client: Submitting application 3 to
> ResourceManager
> 15/12/17 22:22:15 INFO impl.YarnClientImpl: Submitted application
> application_1450383657924_0003
> 15/12/17 22:22:16 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: ACCEPTED)
> 15/12/17 22:22:16 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: default
> start time: 1450383735512
> final status: UNDEFINED
> tracking URL:
> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
> user: hduser
> 15/12/17 22:22:17 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: ACCEPTED)
> 15/12/17 22:22:18 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: ACCEPTED)
> 15/12/17 22:22:19 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: ACCEPTED)
> 15/12/17 22:22:20 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: ACCEPTED)
> 15/12/17 22:22:21 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://
> sparkYarnAM@10.0.0.1:34719/user/YarnAM#-313490512])
> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI
> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
> /proxy/application_1450383657924_0003
> 15/12/17 22:22:21 INFO ui.JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/12/17 22:22:21 INFO yarn.Client: Application report for
> application_1450383657924_0003 (state: RUNNING)
> 15/12/17 22:22:21 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 10.0.0.1
> ApplicationMaster RPC port: 0
> queue: default
> start time: 1450383735512
> final status: UNDEFINED
> tracking URL:
> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/
> user: hduser
> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Application
> application_1450383657924_0003 has started running.
> 15/12/17 22:22:21 INFO util.Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33155.
> 15/12/17 22:22:21 INFO netty.NettyBlockTransferService: Server created on
> 33155
> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 15/12/17 22:22:21 INFO storage.BlockManagerMasterEndpoint: Registering
> block manager 10.0.0.1:33155 with 530.0 MB RAM, BlockManagerId(driver,
> 10.0.0.1, 33155)
> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster has disassociated: 10.0.0.1:34719
> 15/12/17 22:22:22 WARN remote.ReliableDeliverySupervisor: Association with
> remote system [akka.tcp://sparkYarnAM@10.0.0.1:34719] has failed, address
> is now gated for [5000] ms. Reason: [Disassociated]
> 15/12/17 22:22:22 INFO scheduler.EventLoggingListener: Logging events to
> hdfs://localhost:54310/spark-log/application_1450383657924_0003
> 15/12/17 22:22:22 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster has disassociated: 10.0.0.1:34719
> 15/12/17 22:22:25 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://
> sparkYarnAM@10.0.0.1:39353/user/YarnAM#1861872417])
> 15/12/17 22:22:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI
> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES ->
> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003),
> /proxy/application_1450383657924_0003
> 15/12/17 22:22:25 INFO ui.JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster has disassociated: 10.0.0.1:39353
> 15/12/17 22:22:30 WARN remote.ReliableDeliverySupervisor: Association with
> remote system [akka.tcp://sparkYarnAM@10.0.0.1:39353] has failed, address
> is now gated for [5000] ms. Reason: [Disassociated]
> 15/12/17 22:22:30 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster has disassociated: 10.0.0.1:39353
> 15/12/17 22:22:30 ERROR cluster.YarnClientSchedulerBackend: Yarn
> application has already exited with state FINISHED!
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/metrics/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/api,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/static,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/environment/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/environment,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/rdd,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/pool/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/pool,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/job/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/job,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/json,null}
> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs,null}
> 15/12/17 22:22:31 INFO ui.SparkUI: Stopped Spark web UI at
> http://10.0.0.1:4040
> 15/12/17 22:22:31 INFO scheduler.DAGScheduler: Stopping DAGScheduler
> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Shutting down
> all executors
> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Asking each
> executor to shut down
> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Stopped
> 15/12/17 22:22:36 INFO spark.MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 15/12/17 22:22:36 INFO storage.MemoryStore: MemoryStore cleared
> 15/12/17 22:22:36 INFO storage.BlockManager: BlockManager stopped
> 15/12/17 22:22:36 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 15/12/17 22:22:36 INFO spark.SparkContext: Successfully stopped
> SparkContext
> 15/12/17 22:22:36 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remote daemon shut down; proceeding with flushing remote transports.
> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 15/12/17 22:22:42 INFO cluster.YarnClientSchedulerBackend:
> SchedulerBackend is ready for scheduling beginning after waiting
> maxRegisteredResourcesWaitingTime: 30000(ms)
> 15/12/17 22:22:42 INFO repl.SparkILoop: Created spark context..
> Spark context available as sc.
> 15/12/17 22:22:49 INFO hive.HiveContext: Initializing execution hive,
> version 1.2.1
> 15/12/17 22:22:49 INFO client.ClientWrapper: Inspected Hadoop version:
> 2.6.0
> 15/12/17 22:22:49 INFO client.ClientWrapper: Loaded
> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
> 15/12/17 22:22:49 INFO metastore.HiveMetaStore: 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 15/12/17 22:22:49 INFO metastore.ObjectStore: ObjectStore, initialize
> called
> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property
> datanucleus.cache.level2 unknown - will be ignored
> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/12/17 22:22:59 INFO metastore.ObjectStore: Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:02 INFO metastore.MetaStoreDirectSql: Using direct SQL,
> underlying DB is DERBY
> 15/12/17 22:23:02 INFO metastore.ObjectStore: Initialized ObjectStore
> 15/12/17 22:23:02 WARN metastore.ObjectStore: Version information not
> found in metastore. hive.metastore.schema.verification is not enabled so
> recording the schema version 1.2.0
> 15/12/17 22:23:02 WARN metastore.ObjectStore: Failed to get database
> default, returning NoSuchObjectException
> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added admin role in
> metastore
> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added public role in
> metastore
> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: No user is added in admin
> role, since config is empty
> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_all_databases
> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
> cmd=get_all_databases
> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_functions:
> db=default pat=*
> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_functions:
> db=default pat=*
> 15/12/17 22:23:03 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:09 INFO session.SessionState: Created local directory:
> /tmp/eae419f8-7dad-417b-bc3e-77c048c88337_resources
> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
> 15/12/17 22:23:09 INFO session.SessionState: Created local directory:
> /tmp/hduser/eae419f8-7dad-417b-bc3e-77c048c88337
> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory:
> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337/_tmp_space.db
> 15/12/17 22:23:09 INFO hive.HiveContext: default warehouse location is
> /user/hive/warehouse
> 15/12/17 22:23:09 INFO hive.HiveContext: Initializing
> HiveMetastoreConnection version 1.2.1 using Spark classes.
> 15/12/17 22:23:09 INFO client.ClientWrapper: Inspected Hadoop version:
> 2.6.0
> 15/12/17 22:23:09 INFO client.ClientWrapper: Loaded
> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
> 15/12/17 22:23:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/12/17 22:23:10 INFO metastore.HiveMetaStore: 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 15/12/17 22:23:10 INFO metastore.ObjectStore: ObjectStore, initialize
> called
> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property
> datanucleus.cache.level2 unknown - will be ignored
> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/12/17 22:23:11 INFO metastore.ObjectStore: Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:12 INFO DataNucleus.Query: Reading in results for query
> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used
> is closing
> 15/12/17 22:23:12 INFO metastore.MetaStoreDirectSql: Using direct SQL,
> underlying DB is DERBY
> 15/12/17 22:23:12 INFO metastore.ObjectStore: Initialized ObjectStore
> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added admin role in
> metastore
> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added public role in
> metastore
> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: No user is added in admin
> role, since config is empty
> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_all_databases
> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr
> cmd=get_all_databases
> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_functions:
> db=default pat=*
> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_functions:
> db=default pat=*
> 15/12/17 22:23:13 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/12/17 22:23:18 INFO session.SessionState: Created local directory:
> /tmp/7f892b9d-38ef-4fab-bc44-c1afaf07f50f_resources
> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
> 15/12/17 22:23:18 INFO session.SessionState: Created local directory:
> /tmp/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f
> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory:
> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f/_tmp_space.db
> 15/12/17 22:23:18 INFO repl.SparkILoop: Created sql context (with Hive
> support)..
> SQL context available as sqlContext.
>
> scala>
>
> When I am running the following code:
> val data = sc.wholeTextFiles("hdfs://localhost:54310/spark-log/*")
>
> I am getting exception:
> java.lang.IllegalStateException: Cannot call methods on a stopped
> SparkContext
> at org.apache.spark.SparkContext.org
> $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
> at
> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063)
> at
> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076)
> at
> org.apache.spark.SparkContext.wholeTextFiles$default$2(SparkContext.scala:864)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:21)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:32)
> at $iwC$$iwC$$iwC.<init>(<console>:34)
> at $iwC$$iwC.<init>(<console>:36)
> at $iwC.<init>(<console>:38)
> at <init>(<console>:40)
> at .<init>(<console>:44)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
> at org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> *What is wrong here? how should I configure spark to use yarn and hive
> correctly?*
>
> Eran
>
>