[jira] [Commented] (AIRFLOW-3772) Sagemaker operators return unserializable value for XCom

["Oliver Meyer (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361746#comment-17361746 ] 

Oliver Meyer commented on AIRFLOW-3772:
---------------------------------------

I have opened a GitHub issue for this with a suggested fix - see [here|https://github.com/apache/airflow/issues/16386]. [~sarita] and others, would the solution I suggest there work for you? I'll try to open a PR in the next few days.

> Sagemaker operators return unserializable value for XCom
> --------------------------------------------------------
>
>                 Key: AIRFLOW-3772
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3772
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib
>    Affects Versions: 1.10.2
>            Reporter: Daniel Huang
>            Assignee: Daniel Huang
>            Priority: Minor
>
> Sagemaker operators all return a dictionary of metadata in their {{execute}} methods, which in turn, Airflow automatically pushes to XCom. However, the dictionaries include at least one {{datetime}} object. For example, the [SagemakerTransformOperator|https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/operators/sagemaker_transform_operator.py#L117-L124] includes [SageMaker.Client.describe_model|https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.describe_model], which has a {{creationTime}} value.
> This works fine when XCom pickling is enabled (via {{enable_xcom_pickling}}), but if it isn't, it results in a "TypeError: Object of type 'datetime' is not JSON serializable" error. Given that XCom pickling is going to be deprecated in the future, we should make sure the operators return a JSON serializable value.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)