[jira] [Created] (AIRFLOW-3772) Sagemaker operators return unserializable value for XCom

["Daniel Huang (JIRA)" <ji...@apache.org>]

Daniel Huang created AIRFLOW-3772:
-------------------------------------

             Summary: Sagemaker operators return unserializable value for XCom
                 Key: AIRFLOW-3772
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3772
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib
    Affects Versions: 1.10.2
            Reporter: Daniel Huang
            Assignee: Daniel Huang


Sagemaker operators all return a dictionary of metadata in their {{execute}} methods, which in turn, Airflow automatically pushes to XCom. However, the dictionaries include at least one {{datetime}} object. For example, the [SagemakerTransformOperator|https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/operators/sagemaker_transform_operator.py#L117-L124] includes [SageMaker.Client.describe_model|https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.describe_model], which has a {{creationTime}} value.

This works fine when XCom pickling is enabled (via {{enable_xcom_pickling}}), but if it isn't, it results in a "TypeError: Object of type 'datetime' is not JSON serializable" error. Given that XCom pickling is going to be deprecated in the future, we should make sure the operators return a JSON serializable value.
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)