[jira] [Work started] (AIRFLOW-5039) xcom_push execution_date setting is broken when core.enable_xcom_pickling=False

["Rodrigo Martins de Oliveira (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-5039 started by Rodrigo Martins de Oliveira.
------------------------------------------------------------
> xcom_push execution_date setting is broken when core.enable_xcom_pickling=False
> -------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5039
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5039
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: xcom
>    Affects Versions: 1.10.2, 1.10.3
>            Reporter: Rodrigo Martins de Oliveira
>            Assignee: Rodrigo Martins de Oliveira
>            Priority: Critical
>
> Function [xcom_push from airflow.models.taskinstance automatically sets the execution_date to XCOM|https://github.com/apache/airflow/blob/7f66d0a6dc13a790b4d2d31694cb50737983f82e/airflow/models/taskinstance.py#L1336], while this is fine when pickling is enabled it just won't work when it isn't as [XCOM will attempt to JSON serialize a `datetime` object|https://github.com/apache/airflow/blob/7f66d0a6dc13a790b4d2d31694cb50737983f82e/airflow/models/xcom.py#L229], which is unsupported.
>  
> This issue is critical as having pickle enabled poses security risks and [in environments such as Google Cloud Composer pickling is disabled for good|[https://cloud.google.com/composer/docs/concepts/airflow-configurations]].
> Attempting to run DAGs with tasks which pushes content to XCOM when `core-enable_xcom_pickling=False` in `airflow.cfg` will result in the following stacktrace (tested in version 1.10.2-composer):
> {quote}
>  TypeError: Object of type 'datetime' is not JSON serializable
>  [2019-07-24 19:14:53,193] \{models.py:1796} ERROR - Object of type 'datetime' is not JSON serializable
>  Traceback (most recent call last):
>  File "/usr/local/lib/airflow/airflow/models.py", line 1668, in _run_raw_task
>  self.xcom_push(key=XCOM_RETURN_KEY, value=result)
>  File "/usr/local/lib/airflow/airflow/models.py", line 2063, in xcom_push
>  execution_date=execution_date or self.execution_date)
>  File "/usr/local/lib/airflow/airflow/utils/db.py", line 73, in wrapper
>  return func(*args, **kwargs)
>  File "/usr/local/lib/airflow/airflow/models.py", line 4785, in set
>  value = json.dumps(value).encode('UTF-8')
>  File "/opt/python3.6/lib/python3.6/json/__init__.py", line 231, in dumps
>  return _default_encoder.encode(obj)
>  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 199, in encode
>  chunks = self.iterencode(o, _one_shot=True)
>  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
>  return _iterencode(o, 0)
>  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 180, in default
>  o.__class__.__name__)
>  TypeError: Object of type 'datetime' is not JSON serializable
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)