You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/05 11:06:13 UTC

[GitHub] [airflow] ghostbody opened a new issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

ghostbody opened a new issue #19423:
URL: https://github.com/apache/airflow/issues/19423


   ### Description
   
   At present, a `set` method in XCom using two steps:
   
   1. delete all XCom with key `(dag_id, execution_date, task_id)`
   2. insert new XCom
   
   However, when the xcom is set frequently, some error will occur, like this:
   
   ```
   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
   ``` 
   
   ### Use case/motivation
   
   
   using `ON DUPLICATE KEY UPDATE` maybe better?
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ghostbody commented on issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

Posted by GitBox <gi...@apache.org>.
ghostbody commented on issue #19423:
URL: https://github.com/apache/airflow/issues/19423#issuecomment-962403884


   > This is not supported by all databases so you need to write specific code for each backend, and still keep the current implementation when nothing better is available (e.g. for SQLite IIRC). Feel free to try it.
   
   reasonable~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ghostbody closed issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

Posted by GitBox <gi...@apache.org>.
ghostbody closed issue #19423:
URL: https://github.com/apache/airflow/issues/19423


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19423:
URL: https://github.com/apache/airflow/issues/19423#issuecomment-961811107


   This is not supported by all databases so you need to write specific code for each backend, and still keep the current implementation when nothing better is available (e.g. for SQLite IIRC). Feel free to try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19423:
URL: https://github.com/apache/airflow/issues/19423#issuecomment-961811107


   This is not supported by all databases so you need to write specific code for each backend, and still keep the current implementation when nothing better is available (e.g. for SQLite IIRC). Feel free to try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #19423: why not use `ON DUPLICATE KEY UPDATE` in XCom.set

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19423:
URL: https://github.com/apache/airflow/issues/19423#issuecomment-961811107


   This is not supported by all databases so you need to write specific code for each backend, and still keep the current implementation when nothing better is available (e.g. for SQLite IIRC). Feel free to try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org