You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/04 16:19:01 UTC
[GitHub] [airflow] dmateusp opened a new issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
dmateusp opened a new issue #12813:
URL: https://github.com/apache/airflow/issues/12813
**Apache Airflow version**: `1.10.13`
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): `v1.15.11-eks`
```
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-065dce", GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean", BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: AWS (EKS)
- **OS** (e.g. from /etc/os-release):
```
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
- **Kernel** (e.g. `uname -a`): `Linux ddac867b589a 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 GNU/Linux`
- **Python**: `Python 3.7.9`
- **Others**: On Docker, built on top of apache/airflow image (FROM apache/airflow)
**What happened**:
`xcom.pull(...)` throws UnicodeDecodeError when the task that produced the xcom is KubernetesPodOperator:
```
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 161, in get_one
return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```
(Full stack trace lower)
Note that this is happening across all of our DAGs when I try to migrate from 1.10.10 to 1.10.13
**How to reproduce it**:
Here are 2 tasks that should reproduce it (that is part of a DAG I use to test K8s features because making an update internally):
```python
bash_echo_1 = KubernetesPodOperator(
task_id="bash_echo_1",
image="bash:4.4",
name="bash-echo-1", # kubernetes pod names do not accept '_'
cmds=[
"bash",
"-c",
(
"mkdir -p /airflow/xcom "
'&& echo \'{"key1":"value1", "key2": "value2"}\' > /airflow/xcom/return.json'
),
], # that's how xcom works for KubernetesPodOperator
# on_failure_callback=alert_opsgenie, # uncomment to test opgenie
do_xcom_push=True,
) # this needs to be set to true for `/airflow/xcom/return.json` to be pushed as an xcom object
bash_echo_2 = KubernetesPodOperator(
task_id="bash_echo_2",
name="bash-echo-2", # kubernetes pod names do not accept '_'
image="bash:4.4",
arguments=[
"echo",
'key1 was: {{ ti.xcom_pull("bash_echo_1")["key1"] }}',
',key2 was: {{ ti.xcom_pull("bash_echo_1")["key2"] }}',
',the entire object was: {{ ti.xcom_pull("bash_echo_1") }}',
],
)
bash_echo_1 >> bash_echo_2
```
<details><summary>stack trace</summary>
```
Process DagFileProcessor324119-Process:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 159, in _run_file_processor
pickle_dags)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1620, in process_file
self._process_dags(dagbag, dags, ti_keys_to_schedule)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1299, in _process_dags
self._process_task_instances(dag, tis_out)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 793, in _process_task_instances
ready_tis = run.update_state(session=session)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 281, in update_state
ready_tis, changed_tis = self._get_ready_tis(scheduleable_tasks, finished_tasks, session)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 340, in _get_ready_tis
session=session):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 659, in are_dependencies_met
session=session):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses
dep_context):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/not_previously_skipped_dep.py", line 58, in _get_dep_statuses
task_ids=parent.task_id, key=XCOM_SKIPMIXIN_KEY
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1564, in xcom_pull
return pull_fn(task_id=task_ids)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 161, in get_one
return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
``` </details>
Note that bash_echo_2 simply does not get scheduled, the stack trace here comes from the scheduler Pod (we run Airflow on KubernetesExecutor)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-1046652476
If someone experience this issue please add reproduce example.
Since issue is reported against 1.10 (which is EOL) + no comments on this issue for months + ephraim was unable to reproduce I think it's safe to close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-773550431
@dmateusp have you tried this on 2.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12813:
URL: https://github.com/apache/airflow/issues/12813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dmateusp commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dmateusp commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-773572381
sorry folks, we didn't attempt to migrate to 2.0 yet!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-830448573
@michaelosthege, I tested with KubernetesPodOperator. I have not tested your case. Can you add `include_prior_dates=True` to the xcom_pull?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dr-Denzy commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
Dr-Denzy commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-826932722
Let's collaborate on this @ephraimbuddy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-829997969
I was not able to reproduce this on master.
Here's my dag:
<details>
<summary>DAG code</summary>
```python
from airflow import DAG
from datetime import datetime
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
default_args = {
'owner': 'Airflow'}
with DAG('dag_pod_operatorxcom',
start_date=datetime(2019, 3, 12),
schedule_interval='@once',
default_args=default_args,
tags=["core", "http"]) as dag:
bash_echo_1 = KubernetesPodOperator(
in_cluster=True,
namespace='airflow',
task_id="bash_echo_1",
image="bash:4.4",
name="bash-echo-1", # kubernetes pod names do not accept '_'
cmds=[
"bash",
"-c",
(
"mkdir -p /airflow/xcom "
'&& echo \'{"key1":"value1", "key2": "value2"}\' > /airflow/xcom/return.json'
),
], # that's how xcom works for KubernetesPodOperator
# on_failure_callback=alert_opsgenie, # uncomment to test opgenie
do_xcom_push=True,
) # this needs to be set to true for `/airflow/xcom/return.json` to be pushed as an xcom object
bash_echo_2 = KubernetesPodOperator(
in_cluster=True,
namespace='airflow',
task_id="bash_echo_2",
name="bash-echo-2", # kubernetes pod names do not accept '_'
image="bash:4.4",
arguments=[
"echo",
'key1 was: {{ ti.xcom_pull("bash_echo_1")["key1"] }}',
',key2 was: {{ ti.xcom_pull("bash_echo_1")["key2"] }}',
',the entire object was: {{ ti.xcom_pull("bash_echo_1") }}',
],
)
bash_echo_1 >> bash_echo_2
```
</details>
Here's logs for first task:
<details>
<summary>Task bash_echo_1 log</summary>
```log
*** Reading remote log from s3://ephraim-airflowtest/logs/dag_pod_operatorxcom/bash_echo_1/2021-04-30T10:05:08.123775+00:00/1.log.
[2021-04-30 10:05:12,625] {base_task_runner.py:62} DEBUG - Planning to run as the user
[2021-04-30 10:05:12,627] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> from DB
[2021-04-30 10:05:12,632] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Instance Not Running' PASSED: True, Task is not in running state.
[2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Instance State' PASSED: True, Task state queued was valid.
[2021-04-30 10:05:12,636] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:12,636] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2021-04-30 10:05:12,636] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2021-04-30 10:05:12,639] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2021-04-30 10:05:12,642] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Pool Slots Available' PASSED: True, ('There are enough open slots in %s to execute the task', 'default_pool')
[2021-04-30 10:05:12,642] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Concurrency' PASSED: True, Task concurrency is not set.
[2021-04-30 10:05:12,642] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:12,642] {taskinstance.py:1068} INFO -
--------------------------------------------------------------------------------
[2021-04-30 10:05:12,642] {taskinstance.py:1069} INFO - Starting attempt 1 of 1
[2021-04-30 10:05:12,642] {taskinstance.py:1070} INFO -
--------------------------------------------------------------------------------
[2021-04-30 10:05:12,651] {taskinstance.py:1089} INFO - Executing <Task(KubernetesPodOperator): bash_echo_1> on 2021-04-30T10:05:08.123775+00:00
[2021-04-30 10:05:12,653] {standard_task_runner.py:52} INFO - Started process 31 to run task
[2021-04-30 10:05:12,655] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'dag_pod_operatorxcom', 'bash_echo_1', '2021-04-30T10:05:08.123775+00:00', '--job-id', '4', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/podxcompod.py', '--cfg-path', '/tmp/tmp50ostpx4', '--error-file', '/tmp/tmpjti1u7br']
[2021-04-30 10:05:12,655] {standard_task_runner.py:77} INFO - Job 4: Subtask bash_echo_1
[2021-04-30 10:05:12,656] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f193bd0b268>]
[2021-04-30 10:05:12,663] {settings.py:208} DEBUG - Setting up DB connection pool (PID 31)
[2021-04-30 10:05:12,664] {settings.py:241} DEBUG - settings.prepare_engine_args(): Using NullPool
[2021-04-30 10:05:12,665] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [None]> from DB
[2021-04-30 10:05:12,675] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:12,688] {logging_mixin.py:104} INFO - Running <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]> on host dagpodoperatorxcombashecho1.c66682c5cfd84c5a92dfa3ed0e5efe5a
[2021-04-30 10:05:12,688] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:12,694] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:12,696] {taskinstance.py:658} DEBUG - Clearing XCom data
[2021-04-30 10:05:12,701] {taskinstance.py:665} DEBUG - XCom data cleared
[2021-04-30 10:05:12,727] {taskinstance.py:1282} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=Airflow
AIRFLOW_CTX_DAG_ID=dag_pod_operatorxcom
AIRFLOW_CTX_TASK_ID=bash_echo_1
AIRFLOW_CTX_EXECUTION_DATE=2021-04-30T10:05:08.123775+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-30T10:05:08.123775+00:00
[2021-04-30 10:05:12,727] {__init__.py:146} DEBUG - Preparing lineage inlets and outlets
[2021-04-30 10:05:12,727] {__init__.py:190} DEBUG - inlets: [], outlets: []
[2021-04-30 10:05:12,727] {kubernetes_pod.py:434} DEBUG - Creating pod for KubernetesPodOperator task bash_echo_1
[2021-04-30 10:05:12,728] {kubernetes_pod.py:491} DEBUG - Adding xcom sidecar to task bash_echo_1
[2021-04-30 10:05:12,735] {rest.py:228} DEBUG - response body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"8373"},"items":[]}
[2021-04-30 10:05:12,736] {kubernetes_pod.py:367} INFO - creating pod with labels {'dag_id': 'dag_pod_operatorxcom', 'task_id': 'bash_echo_1', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'try_number': '1'} and launcher <airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at 0x7f18d5cb5ba8>
[2021-04-30 10:05:12,736] {kubernetes_pod.py:504} DEBUG - Adding KubernetesPodOperator labels to pod before launch for task bash_echo_1
[2021-04-30 10:05:12,746] {kubernetes_pod.py:518} DEBUG - Starting pod:
api_version: v1
kind: Pod
metadata:
annotations: {}
cluster_name: null
creation_timestamp: null
deletion_grace_period_seconds: null
deletion_timestamp: null
finalizers: null
generate_name: null
generation: null
initializers: null
labels:
airflow_version: 2.1.0.dev0
dag_id: dag_pod_operatorxcom
execution_date: 2021-04-30T100508.1237750000-f80bbc1dc
kubernetes_pod_operator: 'True'
task_id: bash_echo_1
try_number: '1'
managed_fields: null
name: bash-echo-1.741ec5fc537942479902c03db8bae9c9
namespace: airflow
owner_references: null
resource_version: null
self_link: null
uid: null
spec:
active_deadline_seconds: null
affinity:
node_affinity: null
pod_affinity: null
pod_anti_affinity: null
automount_service_account_token: null
containers:
- args: []
command:
- bash
- -c
- 'mkdir -p /airflow/xcom && echo ''{"key1":"value1", "key2": "value2"}'' > /airflow/xcom/return.json'
env: []
env_from: []
image: bash:4.4
image_pull_policy: IfNotPresent
lifecycle: null
liveness_probe: null
name: base
ports: []
readiness_probe: null
resources: {}
security_context: null
stdin: null
stdin_once: null
termination_message_path: null
termination_message_policy: null
tty: null
volume_devices: null
volume_mounts:
- mount_path: /airflow/xcom
mount_propagation: null
name: xcom
read_only: null
sub_path: null
sub_path_expr: null
working_dir: null
- args: null
command:
- sh
- -c
- trap "exit 0" INT; while true; do sleep 1; done;
env: null
env_from: null
image: alpine
image_pull_policy: null
lifecycle: null
liveness_probe: null
name: airflow-xcom-sidecar
ports: null
readiness_probe: null
resources:
limits: null
requests:
cpu: 1m
security_context: null
stdin: null
stdin_once: null
termination_message_path: null
termination_message_policy: null
tty: null
volume_devices: null
volume_mounts:
- mount_path: /airflow/xcom
mount_propagation: null
name: xcom
read_only: null
sub_path: null
sub_path_expr: null
working_dir: null
dns_config: null
dns_policy: null
enable_service_links: null
host_aliases: null
host_ipc: null
host_network: false
host_pid: null
hostname: null
image_pull_secrets: []
init_containers: []
node_name: null
node_selector: {}
preemption_policy: null
priority: null
priority_class_name: null
readiness_gates: null
restart_policy: Never
runtime_class_name: null
scheduler_name: null
security_context: {}
service_account: null
service_account_name: default
share_process_namespace: null
subdomain: null
termination_grace_period_seconds: null
tolerations: []
volumes:
- aws_elastic_block_store: null
azure_disk: null
azure_file: null
cephfs: null
cinder: null
config_map: null
csi: null
downward_api: null
empty_dir:
medium: null
size_limit: null
fc: null
flex_volume: null
flocker: null
gce_persistent_disk: null
git_repo: null
glusterfs: null
host_path: null
iscsi: null
name: xcom
nfs: null
persistent_volume_claim: null
photon_persistent_disk: null
portworx_volume: null
projected: null
quobyte: null
rbd: null
scale_io: null
secret: null
storageos: null
vsphere_volume: null
status: null
[2021-04-30 10:05:12,747] {pod_launcher.py:86} DEBUG - Pod Creation Request:
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {},
"labels": {
"dag_id": "dag_pod_operatorxcom",
"task_id": "bash_echo_1",
"execution_date": "2021-04-30T100508.1237750000-f80bbc1dc",
"try_number": "1",
"airflow_version": "2.1.0.dev0",
"kubernetes_pod_operator": "True"
},
"name": "bash-echo-1.741ec5fc537942479902c03db8bae9c9",
"namespace": "airflow"
},
"spec": {
"affinity": {},
"containers": [
{
"args": [],
"command": [
"bash",
"-c",
"mkdir -p /airflow/xcom && echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' > /airflow/xcom/return.json"
],
"env": [],
"envFrom": [],
"image": "bash:4.4",
"imagePullPolicy": "IfNotPresent",
"name": "base",
"ports": [],
"resources": {},
"volumeMounts": [
{
"mountPath": "/airflow/xcom",
"name": "xcom"
}
]
},
{
"command": [
"sh",
"-c",
"trap \"exit 0\" INT; while true; do sleep 1; done;"
],
"image": "alpine",
"name": "airflow-xcom-sidecar",
"resources": {
"requests": {
"cpu": "1m"
}
},
"volumeMounts": [
{
"mountPath": "/airflow/xcom",
"name": "xcom"
}
]
}
],
"hostNetwork": false,
"imagePullSecrets": [],
"initContainers": [],
"nodeSelector": {},
"restartPolicy": "Never",
"securityContext": {},
"serviceAccountName": "default",
"tolerations": [],
"volumes": [
{
"emptyDir": {},
"name": "xcom"
}
]
}
}
[2021-04-30 10:05:12,750] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8374","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\
u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"
affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","qosClass":"Burstable"}}
[2021-04-30 10:05:12,751] {pod_launcher.py:91} DEBUG - Pod Creation Response: {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': datetime.datetime(2021, 4, 30, 10, 5, 12, tzinfo=tzlocal()),
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'airflow_version': '2.1.0.dev0',
'dag_id': 'dag_pod_operatorxcom',
'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc',
'kubernetes_pod_operator': 'True',
'task_id': 'bash_echo_1',
'try_number': '1'},
'managed_fields': [{'api_version': 'v1',
'fields': None,
'manager': 'OpenAPI-Generator',
'operation': 'Update',
'time': datetime.datetime(2021, 4, 30, 10, 5, 12, tzinfo=tzlocal())}],
'name': 'bash-echo-1.741ec5fc537942479902c03db8bae9c9',
'namespace': 'airflow',
'owner_references': None,
'resource_version': '8374',
'self_link': None,
'uid': '688b6de8-e99b-4caa-894a-750480ead1c8'},
'spec': {'active_deadline_seconds': None,
'affinity': {'node_affinity': None,
'pod_affinity': None,
'pod_anti_affinity': None},
'automount_service_account_token': None,
'containers': [{'args': None,
'command': ['bash',
'-c',
'mkdir -p /airflow/xcom && echo '
'\'{"key1":"value1", "key2": "value2"}\' '
'> /airflow/xcom/return.json'],
'env': None,
'env_from': None,
'image': 'bash:4.4',
'image_pull_policy': 'IfNotPresent',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': None,
'readiness_probe': None,
'resources': {'limits': None, 'requests': None},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/airflow/xcom',
'mount_propagation': None,
'name': 'xcom',
'read_only': None,
'sub_path': None,
'sub_path_expr': None},
{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'default-token-f546n',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None},
{'args': None,
'command': ['sh',
'-c',
'trap "exit 0" INT; while true; do sleep '
'1; done;'],
'env': None,
'env_from': None,
'image': 'alpine',
'image_pull_policy': 'Always',
'lifecycle': None,
'liveness_probe': None,
'name': 'airflow-xcom-sidecar',
'ports': None,
'readiness_probe': None,
'resources': {'limits': None,
'requests': {'cpu': '1m'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/airflow/xcom',
'mount_propagation': None,
'name': 'xcom',
'read_only': None,
'sub_path': None,
'sub_path_expr': None},
{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'default-token-f546n',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': 'ClusterFirst',
'enable_service_links': True,
'host_aliases': None,
'host_ipc': None,
'host_network': None,
'host_pid': None,
'hostname': None,
'image_pull_secrets': None,
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': 'PreemptLowerPriority',
'priority': 0,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': 'default-scheduler',
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': None,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': 'default',
'service_account_name': 'default',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': 30,
'tolerations': [{'effect': 'NoExecute',
'key': 'node.kubernetes.io/not-ready',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None},
{'effect': 'NoExecute',
'key': 'node.kubernetes.io/unreachable',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None}],
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': {'medium': None, 'size_limit': None},
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'xcom',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None},
{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'default-token-f546n',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': {'default_mode': 420,
'items': None,
'optional': None,
'secret_name': 'default-token-f546n'},
'storageos': None,
'vsphere_volume': None}]},
'status': {'conditions': None,
'container_statuses': None,
'host_ip': None,
'init_container_statuses': None,
'message': None,
'nominated_node_name': None,
'phase': 'Pending',
'pod_ip': None,
'qos_class': 'Burstable',
'reason': None,
'start_time': None}}
[2021-04-30 10:05:12,761] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8375","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\
u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-py
thon-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"qosClass":"Burstable"}}
[2021-04-30 10:05:12,762] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending
[2021-04-30 10:05:12,762] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9
[2021-04-30 10:05:13,767] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent
"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"Preempt
LowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}
},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:13,768] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending
[2021-04-30 10:05:13,768] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9
[2021-04-30 10:05:14,772] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent
"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"Preempt
LowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}
},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:14,773] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending
[2021-04-30 10:05:14,773] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9
[2021-04-30 10:05:15,782] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent
"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"Preempt
LowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}
},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:15,783] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending
[2021-04-30 10:05:15,784] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9
[2021-04-30 10:05:16,787] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessage
Path":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute
","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,
"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:16,789] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Running
[2021-04-30 10:05:17,693] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:17,699] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:17,700] {base_job.py:219} DEBUG - [heartbeat]
[2021-04-30 10:05:17,801] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessage
Path":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute
","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,
"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:17,804] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessage
Path":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute
","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,
"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:17,816] {pod_launcher.py:282} INFO - Running command... cat /airflow/xcom/return.json
[2021-04-30 10:05:17,861] {pod_launcher.py:282} INFO - Running command... kill -s SIGINT 1
[2021-04-30 10:05:18,002] {pod_launcher.py:165} INFO - {"key1":"value1", "key2": "value2"}
[2021-04-30 10:05:18,004] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:me
ssage":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessage
Path":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute
","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,
"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:18,006] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Running
[2021-04-30 10:05:18,006] {pod_launcher.py:168} INFO - Pod bash-echo-1.741ec5fc537942479902c03db8bae9c9 has state running
[2021-04-30 10:05:20,012] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8397","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:18Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:re
ason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/term
ination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSec
onds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:15Z","finishedAt":"2021-04-30T10:05:17Z","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8d
c189f216d67649"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":false},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:20,015] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Succeeded
[2021-04-30 10:05:20,015] {pod_launcher.py:302} INFO - Event with job id bash-echo-1.741ec5fc537942479902c03db8bae9c9 Succeeded
[2021-04-30 10:05:20,018] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8397","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requ
ests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:18Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:re
ason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/term
ination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSec
onds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:15Z","finishedAt":"2021-04-30T10:05:17Z","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8d
c189f216d67649"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":false},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}}
[2021-04-30 10:05:20,020] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Succeeded
[2021-04-30 10:05:20,021] {pod_launcher.py:302} INFO - Event with job id bash-echo-1.741ec5fc537942479902c03db8bae9c9 Succeeded
[2021-04-30 10:05:20,032] {__init__.py:107} DEBUG - Lineage called with inlets: [], outlets: []
[2021-04-30 10:05:20,033] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:20,039] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:20,040] {taskinstance.py:1191} INFO - Marking task as SUCCESS. dag_id=dag_pod_operatorxcom, task_id=bash_echo_1, execution_date=20210430T100508, start_date=20210430T100512, end_date=20210430T100520
[2021-04-30 10:05:20,040] {taskinstance.py:1887} DEBUG - Task Duration set to 7.404366
[2021-04-30 10:05:20,060] {dagrun.py:491} DEBUG - number of tis tasks for <DagRun dag_pod_operatorxcom @ 2021-04-30 10:05:08.123775+00:00: manual__2021-04-30T10:05:08.123775+00:00, externally triggered: True>: 2 task(s)
[2021-04-30 10:05:20,060] {dagrun.py:506} DEBUG - number of scheduleable tasks for <DagRun dag_pod_operatorxcom @ 2021-04-30 10:05:08.123775+00:00: manual__2021-04-30T10:05:08.123775+00:00, externally triggered: True>: 1 task(s)
[2021-04-30 10:05:20,061] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30 10:05:08.123775+00:00 [None]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2021-04-30 10:05:20,061] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30 10:05:08.123775+00:00 [None]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2021-04-30 10:05:20,061] {taskinstance.py:877} DEBUG - Dependencies all met for <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30 10:05:08.123775+00:00 [None]>
[2021-04-30 10:05:20,063] {taskinstance.py:1245} INFO - 1 downstream tasks scheduled from follow-on schedule check
[2021-04-30 10:05:20,063] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
[2021-04-30 10:05:20,077] {local_task_job.py:152} INFO - Task exited with return code 0
[2021-04-30 10:05:20,077] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:20,084] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_1 2021-04-30T10:05:08.123775+00:00 [success]>
```
</details>
And logs for the second task:
<details>
<summary>bash echo 2 task</summary>
```log
*** Reading remote log from s3://ephraim-airflowtest/logs/dag_pod_operatorxcom/bash_echo_2/2021-04-30T10:05:08.123775+00:00/1.log.
[2021-04-30 10:05:24,707] {base_task_runner.py:62} DEBUG - Planning to run as the user
[2021-04-30 10:05:24,708] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> from DB
[2021-04-30 10:05:24,714] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Instance Not Running' PASSED: True, Task is not in running state.
[2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Instance State' PASSED: True, Task state queued was valid.
[2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2021-04-30 10:05:24,718] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2021-04-30 10:05:24,718] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:24,718] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Task Concurrency' PASSED: True, Task concurrency is not set.
[2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Pool Slots Available' PASSED: True, ('There are enough open slots in %s to execute the task', 'default_pool')
[2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2021-04-30 10:05:24,729] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [queued]>
[2021-04-30 10:05:24,729] {taskinstance.py:1068} INFO -
--------------------------------------------------------------------------------
[2021-04-30 10:05:24,729] {taskinstance.py:1069} INFO - Starting attempt 1 of 1
[2021-04-30 10:05:24,729] {taskinstance.py:1070} INFO -
--------------------------------------------------------------------------------
[2021-04-30 10:05:24,734] {taskinstance.py:1089} INFO - Executing <Task(KubernetesPodOperator): bash_echo_2> on 2021-04-30T10:05:08.123775+00:00
[2021-04-30 10:05:24,736] {standard_task_runner.py:52} INFO - Started process 31 to run task
[2021-04-30 10:05:24,739] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'dag_pod_operatorxcom', 'bash_echo_2', '2021-04-30T10:05:08.123775+00:00', '--job-id', '5', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/podxcompod.py', '--cfg-path', '/tmp/tmp8kwa7j0v', '--error-file', '/tmp/tmpgcegf545']
[2021-04-30 10:05:24,739] {standard_task_runner.py:77} INFO - Job 5: Subtask bash_echo_2
[2021-04-30 10:05:24,740] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f961e529268>]
[2021-04-30 10:05:24,747] {settings.py:208} DEBUG - Setting up DB connection pool (PID 31)
[2021-04-30 10:05:24,747] {settings.py:241} DEBUG - settings.prepare_engine_args(): Using NullPool
[2021-04-30 10:05:24,749] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [None]> from DB
[2021-04-30 10:05:24,759] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:24,771] {logging_mixin.py:104} INFO - Running <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]> on host dagpodoperatorxcombashecho2.0b7976cfc058439da85e87d672944d13
[2021-04-30 10:05:24,771] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:24,777] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:24,779] {taskinstance.py:658} DEBUG - Clearing XCom data
[2021-04-30 10:05:24,784] {taskinstance.py:665} DEBUG - XCom data cleared
[2021-04-30 10:05:24,827] {taskinstance.py:1282} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=Airflow
AIRFLOW_CTX_DAG_ID=dag_pod_operatorxcom
AIRFLOW_CTX_TASK_ID=bash_echo_2
AIRFLOW_CTX_EXECUTION_DATE=2021-04-30T10:05:08.123775+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-30T10:05:08.123775+00:00
[2021-04-30 10:05:24,827] {__init__.py:146} DEBUG - Preparing lineage inlets and outlets
[2021-04-30 10:05:24,827] {__init__.py:190} DEBUG - inlets: [], outlets: []
[2021-04-30 10:05:24,828] {kubernetes_pod.py:434} DEBUG - Creating pod for KubernetesPodOperator task bash_echo_2
[2021-04-30 10:05:24,835] {rest.py:228} DEBUG - response body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"8415"},"items":[]}
[2021-04-30 10:05:24,836] {kubernetes_pod.py:367} INFO - creating pod with labels {'dag_id': 'dag_pod_operatorxcom', 'task_id': 'bash_echo_2', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'try_number': '1'} and launcher <airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at 0x7f95b8b76080>
[2021-04-30 10:05:24,836] {kubernetes_pod.py:504} DEBUG - Adding KubernetesPodOperator labels to pod before launch for task bash_echo_2
[2021-04-30 10:05:24,841] {kubernetes_pod.py:518} DEBUG - Starting pod:
api_version: v1
kind: Pod
metadata:
annotations: {}
cluster_name: null
creation_timestamp: null
deletion_grace_period_seconds: null
deletion_timestamp: null
finalizers: null
generate_name: null
generation: null
initializers: null
labels:
airflow_version: 2.1.0.dev0
dag_id: dag_pod_operatorxcom
execution_date: 2021-04-30T100508.1237750000-f80bbc1dc
kubernetes_pod_operator: 'True'
task_id: bash_echo_2
try_number: '1'
managed_fields: null
name: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55
namespace: airflow
owner_references: null
resource_version: null
self_link: null
uid: null
spec:
active_deadline_seconds: null
affinity:
node_affinity: null
pod_affinity: null
pod_anti_affinity: null
automount_service_account_token: null
containers:
- args:
- echo
- 'key1 was: value1'
- ',key2 was: value2'
- ',the entire object was: {''key1'': ''value1'', ''key2'': ''value2''}'
command: []
env: []
env_from: []
image: bash:4.4
image_pull_policy: IfNotPresent
lifecycle: null
liveness_probe: null
name: base
ports: []
readiness_probe: null
resources: {}
security_context: null
stdin: null
stdin_once: null
termination_message_path: null
termination_message_policy: null
tty: null
volume_devices: null
volume_mounts: []
working_dir: null
dns_config: null
dns_policy: null
enable_service_links: null
host_aliases: null
host_ipc: null
host_network: false
host_pid: null
hostname: null
image_pull_secrets: []
init_containers: []
node_name: null
node_selector: {}
preemption_policy: null
priority: null
priority_class_name: null
readiness_gates: null
restart_policy: Never
runtime_class_name: null
scheduler_name: null
security_context: {}
service_account: null
service_account_name: default
share_process_namespace: null
subdomain: null
termination_grace_period_seconds: null
tolerations: []
volumes: []
status: null
[2021-04-30 10:05:24,842] {pod_launcher.py:86} DEBUG - Pod Creation Request:
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {},
"labels": {
"dag_id": "dag_pod_operatorxcom",
"task_id": "bash_echo_2",
"execution_date": "2021-04-30T100508.1237750000-f80bbc1dc",
"try_number": "1",
"airflow_version": "2.1.0.dev0",
"kubernetes_pod_operator": "True"
},
"name": "bash-echo-2.c3a98f36477a43f598c786a3cb2efc55",
"namespace": "airflow"
},
"spec": {
"affinity": {},
"containers": [
{
"args": [
"echo",
"key1 was: value1",
",key2 was: value2",
",the entire object was: {'key1': 'value1', 'key2': 'value2'}"
],
"command": [],
"env": [],
"envFrom": [],
"image": "bash:4.4",
"imagePullPolicy": "IfNotPresent",
"name": "base",
"ports": [],
"resources": {},
"volumeMounts": []
}
],
"hostNetwork": false,
"imagePullSecrets": [],
"initContainers": [],
"nodeSelector": {},
"restartPolicy": "Never",
"securityContext": {},
"serviceAccountName": "default",
"tolerations": [],
"volumes": []
}
}
[2021-04-30 10:05:24,844] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8416","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":
[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","qosClass":"BestEffort"}}
[2021-04-30 10:05:24,846] {pod_launcher.py:91} DEBUG - Pod Creation Response: {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': datetime.datetime(2021, 4, 30, 10, 5, 24, tzinfo=tzlocal()),
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'airflow_version': '2.1.0.dev0',
'dag_id': 'dag_pod_operatorxcom',
'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc',
'kubernetes_pod_operator': 'True',
'task_id': 'bash_echo_2',
'try_number': '1'},
'managed_fields': [{'api_version': 'v1',
'fields': None,
'manager': 'OpenAPI-Generator',
'operation': 'Update',
'time': datetime.datetime(2021, 4, 30, 10, 5, 24, tzinfo=tzlocal())}],
'name': 'bash-echo-2.c3a98f36477a43f598c786a3cb2efc55',
'namespace': 'airflow',
'owner_references': None,
'resource_version': '8416',
'self_link': None,
'uid': '4669d35e-6cfe-4863-bdc3-4fdba76ce6b4'},
'spec': {'active_deadline_seconds': None,
'affinity': {'node_affinity': None,
'pod_affinity': None,
'pod_anti_affinity': None},
'automount_service_account_token': None,
'containers': [{'args': ['echo',
'key1 was: value1',
',key2 was: value2',
",the entire object was: {'key1': 'value1', "
"'key2': 'value2'}"],
'command': None,
'env': None,
'env_from': None,
'image': 'bash:4.4',
'image_pull_policy': 'IfNotPresent',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': None,
'readiness_probe': None,
'resources': {'limits': None, 'requests': None},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'default-token-f546n',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': 'ClusterFirst',
'enable_service_links': True,
'host_aliases': None,
'host_ipc': None,
'host_network': None,
'host_pid': None,
'hostname': None,
'image_pull_secrets': None,
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': 'PreemptLowerPriority',
'priority': 0,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': 'default-scheduler',
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': None,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': 'default',
'service_account_name': 'default',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': 30,
'tolerations': [{'effect': 'NoExecute',
'key': 'node.kubernetes.io/not-ready',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None},
{'effect': 'NoExecute',
'key': 'node.kubernetes.io/unreachable',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None}],
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'default-token-f546n',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': {'default_mode': 420,
'items': None,
'optional': None,
'secret_name': 'default-token-f546n'},
'storageos': None,
'vsphere_volume': None}]},
'status': {'conditions': None,
'container_statuses': None,
'host_ip': None,
'init_container_statuses': None,
'message': None,
'nominated_node_name': None,
'phase': 'Pending',
'pod_ip': None,
'qos_class': 'BestEffort',
'reason': None,
'start_time': None}}
[2021-04-30 10:05:24,851] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8419","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash
:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"typ
e":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:24,853] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Pending
[2021-04-30 10:05:24,853] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55
[2021-04-30 10:05:25,861] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8419","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash
:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"typ
e":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:25,863] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Pending
[2021-04-30 10:05:25,863] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55
[2021-04-30 10:05:26,866] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"defaul
t-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionP
olicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker
.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:26,868] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded
[2021-04-30 10:05:26,868] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded
[2021-04-30 10:05:26,871] {pod_launcher.py:148} INFO - key1 was: value1 ,key2 was: value2 ,the entire object was: {'key1': 'value1', 'key2': 'value2'}
[2021-04-30 10:05:27,881] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"defaul
t-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionP
olicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker
.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:27,884] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"defaul
t-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionP
olicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker
.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:27,885] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded
[2021-04-30 10:05:27,885] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded
[2021-04-30 10:05:27,887] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"
f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"defaul
t-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionP
olicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker
.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}}
[2021-04-30 10:05:27,888] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded
[2021-04-30 10:05:27,888] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded
[2021-04-30 10:05:27,888] {__init__.py:107} DEBUG - Lineage called with inlets: [], outlets: []
[2021-04-30 10:05:27,889] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:27,894] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]>
[2021-04-30 10:05:27,895] {taskinstance.py:1191} INFO - Marking task as SUCCESS. dag_id=dag_pod_operatorxcom, task_id=bash_echo_2, execution_date=20210430T100508, start_date=20210430T100524, end_date=20210430T100527
[2021-04-30 10:05:27,895] {taskinstance.py:1887} DEBUG - Task Duration set to 3.177347
[2021-04-30 10:05:27,912] {dagrun.py:491} DEBUG - number of tis tasks for <DagRun dag_pod_operatorxcom @ 2021-04-30 10:05:08.123775+00:00: manual__2021-04-30T10:05:08.123775+00:00, externally triggered: True>: 0 task(s)
[2021-04-30 10:05:27,913] {taskinstance.py:1245} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-04-30 10:05:27,913] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
[2021-04-30 10:05:27,918] {local_task_job.py:152} INFO - Task exited with return code 0
[2021-04-30 10:05:27,918] {taskinstance.py:595} DEBUG - Refreshing TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [running]> from DB
[2021-04-30 10:05:27,926] {taskinstance.py:630} DEBUG - Refreshed TaskInstance <TaskInstance: dag_pod_operatorxcom.bash_echo_2 2021-04-30T10:05:08.123775+00:00 [success]>
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-738876301
@dimberman ? Does it ring a bell ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-773550431
@dmateusp have you tried this on 2.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecwootten commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
ecwootten commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-894180554
I wonder if this might be a wider problem than just the KubernetesPod/Python operators. I have seen this problem on 1.10.12 with a custom operator (derived from BaseOperator) - but it happens only very intermittently.
Our production cluster (AWS ECS, using docker + CeleryExecutor) has been running for ~11 months without issue, and I just saw this bug for the first time. A single xcom JSON couldn't be loaded, and that stopped all DAGs from making any progress. Deleting the relevant record from the xcom table restored everything to working order.
During development (running locally with docker + SequentialExecutor) the bug happened a few times. It seemed to be more likely to happen when the system was under heavy load, but I've not really got enough data to be sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelosthege commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
michaelosthege commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-821833783
We're getting the same error in Airflow 2.0.1 and it happens with xcom produced by the `PythonOperator` too.
Tasks that try to `xcom_pull` upstream results that were created **before** the 1.0.14 → 2.0.1 migration break (see below).
Pulling upstream XCom results that were created **after** the 2.0.1 upgrade works just fine.
```
[2021-04-17 14:28:35,344] {taskinstance.py:1455} ERROR - 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/opt/conda/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
return_value = self.execute_callable()
File "/opt/conda/lib/python3.7/site-packages/airflow/operators/python.py", line 128, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/flow.py", line 91, in _run
upstream_result = task_instance.xcom_pull(task_ids=[taskid])[0]
File "/opt/conda/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1960, in xcom_pull
for result in query.with_entities(XCom.task_id, XCom.value)
File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1960, in <dictcomp>
for result in query.with_entities(XCom.task_id, XCom.value)
File "/opt/conda/lib/python3.7/site-packages/airflow/models/xcom.py", line 255, in deserialize_value
return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```
Interestingly the XCom data shows up just fine on the webserver:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelosthege commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
michaelosthege commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-832056544
@ephraimbuddy the tasks were part of the same DAG. (I cleared the downstream one after the 2.0.1 upgrade, but not the upstream task that had run on 1.10.14.)
Therefore it doesn't sound like `include_prior_dates` should make any difference, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-763195298
Does it happen on 2.0.0 too ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-1046652476
If someone experience this issue please add reproduce example.
Since no comments on this issue for months + ephraim was unable to reproduce I think it's safe to close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecwootten edited a comment on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
ecwootten edited a comment on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-1046658608
Since my last comment, we've experienced this bug twice on our production cluster in 6 months of otherwise trouble-free running. So it is highly intermittent, and we have not found any way to reliably reproduce it. It is also a very problematic bug, because it completely disables the DAGs - but I appreciate that 1.10 is a pretty old version now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-829737575
Hi @Dr-Denzy @ephraimbuddy any update on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-826930615
@Dr-Denzy @ephraimbuddy would either of you be interested in this ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blessymoses commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
blessymoses commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-757711168
This issue also results in BranchPythonOperator not working with KubernetesExecutor.
**Apache Airflow version**: 1.10.12
**What happened**:
BranchPythonOperator task succeeds. But it does not schedule the task in the followed branch.
Scheduler throws UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
**Stack Trace**:
Process DagFileProcessor4425-Process:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 159, in _run_file_processor
pickle_dags)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1637, in process_file
self._process_dags(dagbag, dags, ti_keys_to_schedule)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1316, in _process_dags
self._process_task_instances(dag, tis_out)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 793, in _process_task_instances
run.update_state(session=session)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 296, in update_state
session=session)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 659, in are_dependencies_met
session=session):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses
dep_context):
File "/usr/local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/usr/local/lib/python3.7/site-packages/airflow/ti_deps/deps/not_previously_skipped_dep.py", line 58, in _get_dep_statuses
task_ids=parent.task_id, key=XCOM_SKIPMIXIN_KEY
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1564, in xcom_pull
return pull_fn(task_id=task_ids)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/xcom.py", line 165, in get_one
return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-738874084
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecwootten commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
ecwootten commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-1046658608
Since my last comment, we've experienced this bug twice on our production cluster in 6 months of otherwise trouble-free running. So it is highly intermittent, and we have not found any way to reliably reproduce it. It is also a very problematic bug, because it completely disables the DGAs - but I appreciate that 1.10 is a pretty old version now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelosthege commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
michaelosthege commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-830276395
@ephraimbuddy both your tasks ran with Airflow 2.1.0.?
In my case the first task ran with Airflow 1.10.14 and the second with 2.0.1.
They werde PythonOperators and the return value was a simple dictionary.
But I can't tell if that is the same problem as with the KubernetesPodOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dmateusp commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError
Posted by GitBox <gi...@apache.org>.
dmateusp commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-773572381
sorry folks, we didn't attempt to migrate to 2.0 yet!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org