[GitHub] [airflow] potiuk commented on issue #16386: SageMaker operators return non-serializable value

[GitBox <gi...@apache.org>]

potiuk commented on issue #16386:
URL: https://github.com/apache/airflow/issues/16386#issuecomment-1060017364


   > what's the solution here? I am running a SageMakerProcessingOperator Airflow task the processing job executed successfully but I got the error "ERROR - Could not serialize the XCom value into JSON. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config., TypeError: Object of type datetime is not JSON serializable" . How should I get away from this ? I have downstream task depend on success of SageMakerProcessingOperator
   
   a) set `do_xcom_push` to False:  https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html
   
   b) derive your own custom operator from the SageMarkerOperator and return nothing (or converte return value to serializable. Smth like: 
   
   ```
   class MyOperator(SageMakerProcessingOperator):
       def execute():
          ret = super().execute()
         serializable_return = .....  # convert it to serializable
         return (serializable_return)
   ```
   
   c) enable pickle in Airflow configuration https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html?highlight=pickle#enable-xcom-pickling
   
   d) correct it in the operator and contribute it back (this is by far the best idea).
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org