You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Farhan Khan (Jira)" <ji...@apache.org> on 2020/10/20 23:02:00 UTC
[jira] [Updated] (SPARK-33199) Mesos Task Failed when pyFiles and
docker image option used together
[ https://issues.apache.org/jira/browse/SPARK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Farhan Khan updated SPARK-33199:
--------------------------------
Description:
It is a bug related to shell escaping in MesosClusterScheduler. As you can see in this issue SPARK-23941, we are escaping all the options, that lead to escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image option as well. This in turn causes *FileNotFoundErrors* during *py4j.SparkSession.getOrCreate*.
In order to reproduce the bug, you need to have mesos and mesos dispatcher running. When you launch the following program:
{code:bash}
spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class my_python_program.py --py-files /path/to/PySparkInclude.py --name my_python_program --conf spark.mesos.executor.docker.image=myrepo/myimage:mytag /path/to/my_python_program.py 100
{code}
then the task failed with the following output:
{code:bash}
I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"}
I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI '/path/to/my_python_program.py'
I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching '/path/to/my_python_program.py' directly into the sandbox directory
I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py'
W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/my_python_program.py
I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py'
I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI '/path/to/PySparkTestInclude.py'
I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching '/path/to/PySparkTestInclude.py' directly into the sandbox directory
I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/PySparkTestInclude.py
I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs into '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375'
Marked '/' as rslave
W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be downgraded to a non-SSL socket
W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are enabled.
I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath>
I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification in subject alternative name certificate extension.
I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates for outgoing TLS connections.
I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for hostname validation
I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt
W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be downgraded to a non-SSL socket
I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1
I1020 16:26:12.392472 2 executor.cpp:286] Using URL http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor for the executor API endpoint
I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event
I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on 172.16.72.91
I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event
I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CA_FILE'
I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CERT_FILE'
I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment variable 'MESOS_NATIVE_JAVA_LIBRARY'
I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_KEY_FILE'
I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment variable 'PATH'
I1020 16:26:12.400053 8 executor.cpp:722] Starting task driver-20201020162611-0012
I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12
I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event
I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event
Preparing rootfs at '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462'
Marked '/' as rslave
Changing root to /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462
spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91
spark-env: User: nobody
spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode
spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in /opt/spark/conf/metrics.properties
20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1
20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver:
20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi
20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody
20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody
20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to:
20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to:
20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody); groups with view permissions: Set(); users with modify permissions: Set(nobody); groups with modify permissions: Set()
20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on port 37759.
20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker
20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster
20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded
20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB
20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from resource metrics.properties
20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, *.sink.statsd.host=198.51.100.1, *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, applications.sink.servlet.path=/metrics/applications/json, *.sink.statsd.tags=, *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json}
20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator
20/10/20 16:26:18 INFO log: Logging initialized @5603ms to org.sparkproject.jetty.util.log.Slf4jLog
20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b03
20/10/20 16:26:18 INFO Server: Started @5872ms
20/10/20 16:26:18 INFO AbstractConnector: Started ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at http://172.16.72.91:4040
20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498)
at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494)
at org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:494)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
20/10/20 16:26:18 INFO AbstractConnector: Stopped Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at http://172.16.72.91:4040
20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared
20/10/20 16:26:19 INFO BlockManager: BlockManager stopped
20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped
20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running
20/10/20 16:26:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext
20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-07db5724-6031-45d3-8222-c06947f80872
20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f
I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 (pid: 12)
I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event
I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop
{code}
was:
It is a bug related to shell escaping in MesosClusterScheduler. As you can see in this issue SPARK-23941, we are escaping all the options, that lead to escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image option as well. This in turn causes *FileNotFoundErrors* during *py4j.SparkSession.getOrCreate*.
In order to reproduce the bug, you need to have mesos and mesos dispatcher running. When you launch the following program:
{code:bash}
spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class my_python_program.py --py-files /path/to/PySparkInclude.py --name my_python_program --conf spark.mesos.executor.docker.image=myrepo/myimage:mytag /path/to/my_python_program.py 100
{code}
then the task failed with the following output:
{code:bash}
I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"}
I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI '/path/to/my_python_program.py'
I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching '/path/to/my_python_program.py' directly into the sandbox directory
I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/pi_with_include.py'
W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/my_python_program.py
I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/pi_with_include.py'
I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI '/path/to/PySparkTestInclude.py'
I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching '/path/to/PySparkTestInclude.py' directly into the sandbox directory
I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/PySparkTestInclude.py
I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs into '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375'
Marked '/' as rslave
W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be downgraded to a non-SSL socket
W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are enabled.
I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath>
I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification in subject alternative name certificate extension.
I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates for outgoing TLS connections.
I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for hostname validation
I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt
W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be downgraded to a non-SSL socket
I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1
I1020 16:26:12.392472 2 executor.cpp:286] Using URL http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor for the executor API endpoint
I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event
I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on 172.16.72.91
I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event
I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CA_FILE'
I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CERT_FILE'
I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment variable 'MESOS_NATIVE_JAVA_LIBRARY'
I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_KEY_FILE'
I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment variable 'PATH'
I1020 16:26:12.400053 8 executor.cpp:722] Starting task driver-20201020162611-0012
I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12
I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event
I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event
Preparing rootfs at '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462'
Marked '/' as rslave
Changing root to /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462
spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91
spark-env: User: nobody
spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode
spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in /opt/spark/conf/metrics.properties
20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1
20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver:
20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi
20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody
20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody
20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to:
20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to:
20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody); groups with view permissions: Set(); users with modify permissions: Set(nobody); groups with modify permissions: Set()
20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on port 37759.
20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker
20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster
20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded
20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB
20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from resource metrics.properties
20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, *.sink.statsd.host=198.51.100.1, *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, applications.sink.servlet.path=/metrics/applications/json, *.sink.statsd.tags=, *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json}
20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator
20/10/20 16:26:18 INFO log: Logging initialized @5603ms to org.sparkproject.jetty.util.log.Slf4jLog
20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b03
20/10/20 16:26:18 INFO Server: Started @5872ms
20/10/20 16:26:18 INFO AbstractConnector: Started ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark}
20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at http://172.16.72.91:4040
20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498)
at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494)
at org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:494)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
20/10/20 16:26:18 INFO AbstractConnector: Stopped Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at http://172.16.72.91:4040
20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared
20/10/20 16:26:19 INFO BlockManager: BlockManager stopped
20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped
20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running
20/10/20 16:26:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext
20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-07db5724-6031-45d3-8222-c06947f80872
20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f
I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 (pid: 12)
I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event
I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop
{code}
> Mesos Task Failed when pyFiles and docker image option used together
> --------------------------------------------------------------------
>
> Key: SPARK-33199
> URL: https://issues.apache.org/jira/browse/SPARK-33199
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 3.0.1
> Reporter: Farhan Khan
> Priority: Major
>
> It is a bug related to shell escaping in MesosClusterScheduler. As you can see in this issue SPARK-23941, we are escaping all the options, that lead to escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image option as well. This in turn causes *FileNotFoundErrors* during *py4j.SparkSession.getOrCreate*.
> In order to reproduce the bug, you need to have mesos and mesos dispatcher running. When you launch the following program:
> {code:bash}
> spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class my_python_program.py --py-files /path/to/PySparkInclude.py --name my_python_program --conf spark.mesos.executor.docker.image=myrepo/myimage:mytag /path/to/my_python_program.py 100
> {code}
> then the task failed with the following output:
> {code:bash}
> I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"}
> I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI '/path/to/my_python_program.py'
> I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching '/path/to/my_python_program.py' directly into the sandbox directory
> I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py'
> W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/my_python_program.py
> I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py'
> I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI '/path/to/PySparkTestInclude.py'
> I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching '/path/to/PySparkTestInclude.py' directly into the sandbox directory
> I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
> W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/PySparkTestInclude.py
> I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py'
> I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs into '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375'
> Marked '/' as rslave
> W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be downgraded to a non-SSL socket
> W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are enabled.
> I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath>
> I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification in subject alternative name certificate extension.
> I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates for outgoing TLS connections.
> I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for hostname validation
> I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt
> W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be downgraded to a non-SSL socket
> I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1
> I1020 16:26:12.392472 2 executor.cpp:286] Using URL http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor for the executor API endpoint
> I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event
> I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on 172.16.72.91
> I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event
> I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CA_FILE'
> I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CERT_FILE'
> I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment variable 'MESOS_NATIVE_JAVA_LIBRARY'
> I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_KEY_FILE'
> I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment variable 'PATH'
> I1020 16:26:12.400053 8 executor.cpp:722] Starting task driver-20201020162611-0012
> I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12
> I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event
> I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event
> Preparing rootfs at '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462'
> Marked '/' as rslave
> Changing root to /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462
> spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91
> spark-env: User: nobody
> spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode
> spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in /opt/spark/conf/metrics.properties
> 20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1
> 20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
> 20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver:
> 20/10/20 16:26:17 INFO ResourceUtils: ==============================================================
> 20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi
> 20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody
> 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody
> 20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to:
> 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to:
> 20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody); groups with view permissions: Set(); users with modify permissions: Set(nobody); groups with modify permissions: Set()
> 20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on port 37759.
> 20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker
> 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster
> 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded
> 20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB
> 20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from resource metrics.properties
> 20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, *.sink.statsd.host=198.51.100.1, *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, applications.sink.servlet.path=/metrics/applications/json, *.sink.statsd.tags=, *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json}
> 20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator
> 20/10/20 16:26:18 INFO log: Logging initialized @5603ms to org.sparkproject.jetty.util.log.Slf4jLog
> 20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b03
> 20/10/20 16:26:18 INFO Server: Started @5872ms
> 20/10/20 16:26:18 INFO AbstractConnector: Started ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
> 20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark}
> 20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at http://172.16.72.91:4040
> 20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext.
> java.io.FileNotFoundException: File file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534)
> at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498)
> at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494)
> at org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:494)
> at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:238)
> at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> 20/10/20 16:26:18 INFO AbstractConnector: Stopped Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040}
> 20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at http://172.16.72.91:4040
> 20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
> 20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared
> 20/10/20 16:26:19 INFO BlockManager: BlockManager stopped
> 20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped
> 20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running
> 20/10/20 16:26:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
> 20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext
> 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
> 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called
> 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-07db5724-6031-45d3-8222-c06947f80872
> 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f
> I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 (pid: 12)
> I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event
> I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org