[GitHub] [airflow] ecwootten commented on issue #12813: Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError

[GitBox <gi...@apache.org>]

ecwootten commented on issue #12813:
URL: https://github.com/apache/airflow/issues/12813#issuecomment-894180554


   I wonder if this might be a wider problem than just the KubernetesPod/Python operators. I have seen this problem on 1.10.12 with a custom operator (derived from BaseOperator) - but it happens only very intermittently. 
   
   Our production cluster (AWS ECS, using docker + CeleryExecutor) has been running for ~11 months without issue, and I just saw this bug for the first time. A single xcom JSON couldn't be loaded, and that stopped all DAGs from making any progress. Deleting the relevant record from the xcom table restored everything to working order.
   
   During development (running locally with docker + SequentialExecutor) the bug happened a few times. It seemed to be more likely to happen when the system was under heavy load, but I've not really got enough data to be sure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org