You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/07/07 19:22:14 UTC
[GitHub] [airflow] Fokko opened a new pull request #5542: [AIRFLOW-4906]
Improve debugging for the SparkSubmitHook
Fokko opened a new pull request #5542: [AIRFLOW-4906] Improve debugging for the SparkSubmitHook
URL: https://github.com/apache/airflow/pull/5542
Make sure you have checked _all_ steps below.
Before:
```
[2019-07-07 18:03:34,465] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [scheduled]>
[2019-07-07 18:03:34,486] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [scheduled]>
[2019-07-07 18:03:34,487] {__init__.py:1353} INFO -
--------------------------------------------------------------------------------
[2019-07-07 18:03:34,488] {__init__.py:1354} INFO - Starting attempt 2 of 2
[2019-07-07 18:03:34,488] {__init__.py:1355} INFO -
--------------------------------------------------------------------------------
[2019-07-07 18:03:34,526] {__init__.py:1374} INFO - Executing <Task(SparkSubmitOperator): compute_pi> on 2019-07-04T00:00:00+00:00
[2019-07-07 18:03:34,530] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'hello_spark', 'compute_pi', '2019-07-04T00:00:00+00:00', '--job_id', '36', '--raw', '-sd', 'DAGS_FOLDER/spark.py', '--cfg_path', '/tmp/tmpviu_vxcj']
[2019-07-07 18:03:36,961] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi [2019-07-07 18:03:36,956] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=15
[2019-07-07 18:03:37,520] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi [2019-07-07 18:03:37,508] {__init__.py:51} INFO - Using executor LocalExecutor
[2019-07-07 18:03:38,325] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi [2019-07-07 18:03:38,323] {__init__.py:305} INFO - Filling up the DagBag from /root/airflow/dags/spark.py
[2019-07-07 18:03:39,236] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi [2019-07-07 18:03:39,234] {cli.py:517} INFO - Running <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [running]> on host hellosparkcomputepi-e6e6c0eba6d248848ebbe6d3102b29d1
[2019-07-07 18:03:39,403] {logging_mixin.py:95} INFO - [2019-07-07 18:03:39,403] {base_hook.py:83} INFO - Using connection to: id: local-spark. Host: k8s://http://192.168.1.113, Port: 8080, Schema: None, Login: None, Password: None, extra: {'spark_home': '/usr/spark', 'deploy-mode': 'cluster'}
[2019-07-07 18:03:39,407] {logging_mixin.py:95} INFO - [2019-07-07 18:03:39,405] {spark_submit_hook.py:295} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']
[2019-07-07 18:03:45,170] {__init__.py:1580} ERROR - Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.5/dist-packages/airflow/contrib/operators/spark_submit_operator.py", line 176, in execute
self._hook.submit(self._application)
File "/usr/local/lib/python3.5/dist-packages/airflow/contrib/hooks/spark_submit_hook.py", line 352, in submit
spark_submit_cmd, returncode
airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
[2019-07-07 18:03:45,175] {__init__.py:1611} INFO - Marking task as FAILED.
[2019-07-07 18:03:45,206] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi /usr/local/lib/python3.5/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
[2019-07-07 18:03:45,207] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi """)
[2019-07-07 18:03:45,209] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi Traceback (most recent call last):
[2019-07-07 18:03:45,210] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/bin/airflow", line 32, in <module>
[2019-07-07 18:03:45,211] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi args.func(args)
[2019-07-07 18:03:45,211] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-07-07 18:03:45,212] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi return f(*args, **kwargs)
[2019-07-07 18:03:45,213] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 523, in run
[2019-07-07 18:03:45,213] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi _run(args, dag, ti)
[2019-07-07 18:03:45,214] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 442, in _run
[2019-07-07 18:03:45,215] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi pool=args.pool,
[2019-07-07 18:03:45,215] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line 73, in wrapper
[2019-07-07 18:03:45,216] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi return func(*args, **kwargs)
[2019-07-07 18:03:45,217] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-07-07 18:03:45,217] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi result = task_copy.execute(context=context)
[2019-07-07 18:03:45,218] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/contrib/operators/spark_submit_operator.py", line 176, in execute
[2019-07-07 18:03:45,219] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi self._hook.submit(self._application)
[2019-07-07 18:03:45,220] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/contrib/hooks/spark_submit_hook.py", line 352, in submit
[2019-07-07 18:03:45,220] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi spark_submit_cmd, returncode
[2019-07-07 18:03:45,221] {base_task_runner.py:101} INFO - Job 36: Subtask compute_pi airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
[2019-07-07 18:03:49,544] {logging_mixin.py:95} INFO - [2019-07-07 18:03:49,542] {jobs.py:2562} INFO - Task exited with return code 1
```
Becomes:
```
[2019-07-07 19:13:08,038] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [queued]>
[2019-07-07 19:13:08,102] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [queued]>
[2019-07-07 19:13:08,105] {__init__.py:1353} INFO -
--------------------------------------------------------------------------------
[2019-07-07 19:13:08,107] {__init__.py:1354} INFO - Starting attempt 3 of 3
[2019-07-07 19:13:08,112] {__init__.py:1355} INFO -
--------------------------------------------------------------------------------
[2019-07-07 19:13:08,209] {__init__.py:1374} INFO - Executing <Task(SparkSubmitOperator): compute_pi> on 2019-07-04T00:00:00+00:00
[2019-07-07 19:13:08,211] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'hello_spark', 'compute_pi', '2019-07-04T00:00:00+00:00', '--job_id', '42', '--raw', '-sd', 'DAGS_FOLDER/spark.py', '--cfg_path', '/tmp/tmpvrp9os2m']
[2019-07-07 19:13:10,715] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi [2019-07-07 19:13:10,715] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=15
[2019-07-07 19:13:10,925] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi [2019-07-07 19:13:10,924] {__init__.py:51} INFO - Using executor LocalExecutor
[2019-07-07 19:13:11,432] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi [2019-07-07 19:13:11,431] {__init__.py:305} INFO - Filling up the DagBag from /root/airflow/dags/spark.py
[2019-07-07 19:13:12,184] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi [2019-07-07 19:13:12,182] {cli.py:517} INFO - Running <TaskInstance: hello_spark.compute_pi 2019-07-04T00:00:00+00:00 [running]> on host hellosparkcomputepi-477f1a3c92534128a7dc03183888b1a8
[2019-07-07 19:13:12,283] {logging_mixin.py:95} INFO - [2019-07-07 19:13:12,283] {base_hook.py:83} INFO - Using connection to: id: local-spark. Host: k8s://http://192.168.1.113, Port: 8080, Schema: None, Login: None, Password: None, extra: {'spark_home': '/usr/spark', 'deploy-mode': 'cluster'}
[2019-07-07 19:13:12,286] {logging_mixin.py:95} INFO - [2019-07-07 19:13:12,284] {spark_submit_hook.py:295} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']
[2019-07-07 19:13:14,983] {logging_mixin.py:95} INFO - [2019-07-07 19:13:14,983] {spark_submit_hook.py:426} INFO - 19/07/07 19:13:14 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS.
[2019-07-07 19:13:16,296] {logging_mixin.py:95} INFO - [2019-07-07 19:13:16,295] {spark_submit_hook.py:426} INFO - log4j:WARN No appenders could be found for logger (io.fabric8.kubernetes.client.Config).
[2019-07-07 19:13:16,297] {logging_mixin.py:95} INFO - [2019-07-07 19:13:16,297] {spark_submit_hook.py:426} INFO - log4j:WARN Please initialize the log4j system properly.
[2019-07-07 19:13:16,298] {logging_mixin.py:95} INFO - [2019-07-07 19:13:16,298] {spark_submit_hook.py:426} INFO - log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2019-07-07 19:13:17,114] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,114] {spark_submit_hook.py:426} INFO - Exception in thread "main" org.apache.spark.SparkException: Must specify the driver container image
[2019-07-07 19:13:17,121] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,121] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep$$anonfun$3.apply(BasicDriverFeatureStep.scala:42)
[2019-07-07 19:13:17,123] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,123] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep$$anonfun$3.apply(BasicDriverFeatureStep.scala:42)
[2019-07-07 19:13:17,128] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,127] {spark_submit_hook.py:426} INFO - at scala.Option.getOrElse(Option.scala:121)
[2019-07-07 19:13:17,130] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,130] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.<init>(BasicDriverFeatureStep.scala:42)
[2019-07-07 19:13:17,132] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,131] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder$$anonfun$$lessinit$greater$default$1$1.apply(KubernetesDriverBuilder.scala:25)
[2019-07-07 19:13:17,135] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,135] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder$$anonfun$$lessinit$greater$default$1$1.apply(KubernetesDriverBuilder.scala:25)
[2019-07-07 19:13:17,138] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,138] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:59)
[2019-07-07 19:13:17,143] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,143] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:110)
[2019-07-07 19:13:17,145] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,144] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
[2019-07-07 19:13:17,146] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,145] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
[2019-07-07 19:13:17,148] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,147] {spark_submit_hook.py:426} INFO - at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
[2019-07-07 19:13:17,151] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,151] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
[2019-07-07 19:13:17,155] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,155] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
[2019-07-07 19:13:17,156] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,156] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
[2019-07-07 19:13:17,158] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,157] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
[2019-07-07 19:13:17,160] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,160] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
[2019-07-07 19:13:17,161] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,161] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
[2019-07-07 19:13:17,162] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,161] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
[2019-07-07 19:13:17,162] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,162] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
[2019-07-07 19:13:17,163] {logging_mixin.py:95} INFO - [2019-07-07 19:13:17,163] {spark_submit_hook.py:426} INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[2019-07-07 19:13:17,575] {__init__.py:1580} ERROR - Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/root/airflow/dags/operators/spark_submit_operator.py", line 176, in execute
self._hook.submit(self._application)
File "/root/airflow/dags/hooks/spark_submit_hook.py", line 352, in submit
spark_submit_cmd, returncode
airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
[2019-07-07 19:13:17,594] {__init__.py:1611} INFO - Marking task as FAILED.
[2019-07-07 19:13:17,626] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi /usr/local/lib/python3.5/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
[2019-07-07 19:13:17,627] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi """)
[2019-07-07 19:13:17,640] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi Traceback (most recent call last):
[2019-07-07 19:13:17,646] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/bin/airflow", line 32, in <module>
[2019-07-07 19:13:17,648] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi args.func(args)
[2019-07-07 19:13:17,649] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-07-07 19:13:17,649] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi return f(*args, **kwargs)
[2019-07-07 19:13:17,651] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 523, in run
[2019-07-07 19:13:17,652] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi _run(args, dag, ti)
[2019-07-07 19:13:17,652] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 442, in _run
[2019-07-07 19:13:17,653] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi pool=args.pool,
[2019-07-07 19:13:17,653] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line 73, in wrapper
[2019-07-07 19:13:17,656] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi return func(*args, **kwargs)
[2019-07-07 19:13:17,657] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/usr/local/lib/python3.5/dist-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-07-07 19:13:17,662] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi result = task_copy.execute(context=context)
[2019-07-07 19:13:17,663] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/root/airflow/dags/operators/spark_submit_operator.py", line 176, in execute
[2019-07-07 19:13:17,664] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi self._hook.submit(self._application)
[2019-07-07 19:13:17,665] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi File "/root/airflow/dags/hooks/spark_submit_hook.py", line 352, in submit
[2019-07-07 19:13:17,668] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi spark_submit_cmd, returncode
[2019-07-07 19:13:17,669] {base_task_runner.py:101} INFO - Job 42: Subtask compute_pi airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'k8s://http://192.168.1.113:8080', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.options.claimName=airflow-dags', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.driver.volumes.persistentVolumeClaim.shared.mount.readOnly=true', '--conf', 'spark.kubernetes.executor.volumes.persistentVolumeClaim.shared.mount.path=/code', '--conf', 'spark.kubernetes.namespace=default', '--name', 'airflow-spark', '--deploy-mode', 'cluster', 'local:///code/pi.py']. Error code is: 1.
[2019-07-07 19:13:18,137] {logging_mixin.py:95} INFO - [2019-07-07 19:13:18,137] {jobs.py:2630} WARNING - State of this instance has been externally set to failed. Taking the poison pill.
[2019-07-07 19:13:18,157] {helpers.py:281} INFO - Sending Signals.SIGTERM to GPID 15
[2019-07-07 19:13:18,238] {helpers.py:263} INFO - Process psutil.Process(pid=15, status='terminated') (15) terminated with exit code -15
[2019-07-07 19:13:18,270] {logging_mixin.py:95} INFO - [2019-07-07 19:13:18,259] {jobs.py:2562} INFO - Task exited with return code 0
```
Otherwise, it would become very hard to find errors in the configuration (in the image in this case).
### Jira
- [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-4906\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4906
- In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-4906\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services