You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/27 12:24:32 UTC
[GitHub] [airflow] gedOHub opened a new issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
gedOHub opened a new issue #13925:
URL: https://github.com/apache/airflow/issues/13925
**Apache Airflow version**: 2.0.0
**Kubernetes version**: -
**Environment**:
- Two servers running docker
- Docker image:` apache/airflow:2.0.0-python3.7`
- Each server contains:
- webserver
- scheduler
- celeny worker 1
- celeny worker 2
- celeny flower
- Broker- redis version `6.0.5`
- DB backend - MySQL version `8.0.20`
**What happened**:
Error message:
```bash
[...] {taskinstance.py:1038} INFO - Executing <Task(BashOperator): xxx.yyy> on ...
[...] {standard_task_runner.py:51} INFO - Started process 4245 to run task
[...] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'zzz', 'xxx.yyy', '...', '--job-id', '8039', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/ccc/ccc.zip', '--cfg-path', '/tmp/tmpq2aaznri']
[...] {standard_task_runner.py:76} INFO - Job 8039: Subtask xxx.yyy
[...] {logging_mixin.py:103} INFO - Running <TaskInstance: zzz.xxx.yyy ... [running]> on host HOST
[...] {taskinstance.py:1396} ERROR - (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'zzz-xxx.can_run_tasks_loca' for key 'rendered_task_instance_fields.PRIMARY'")
[SQL: INSERT INTO rendered_task_instance_fields (dag_id, task_id, execution_date, rendered_fields, k8s_pod_yaml) VALUES (%s, %s, %s, %s, %s)]
[parameters: ('zzz', 'xxx.yyy', datetime.datetime(..., ..., ..., ..., ...), '{"bash_command": "mkdir -p ~/eee && rm -rf ~/eee", "env": null}', 'null')]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/airflow/.local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 255, in execute
self.errorhandler(self, exc, value)
File "/home/airflow/.local/lib/python3.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/home/airflow/.local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 252, in execute
res = self._query(query)
File "/home/airflow/.local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 378, in _query
db.query(q)
File "/home/airflow/.local/lib/python3.7/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
_mysql_exceptions.IntegrityError: (1062, "Duplicate entry 'zzz-xxx.yyy' for key 'rendered_task_instance_fields.PRIMARY'")
```
**What you expected to happen**:
Run DAG without failing
**How to reproduce it**:
Use MySQL backend?
**Anything else we need to know**:
This problem occurs once in a while. I haven't found any pattern.
Maybe this related to https://github.com/apache/airflow/issues/9148 issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-919595272
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #13925:
URL: https://github.com/apache/airflow/issues/13925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-768381953
The fix for #9148 already landed in Master too https://github.com/apache/airflow/commit/9e6b5abea08eb35c48839e33a4518f292ef7564e so not sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
gedOHub commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-772542269
@kaxil is there a known new release date that will have your fixes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-772833316
> @kaxil is there a known new release date that will have your fixes?
The fix was included in 2.0.0 itself, so I am a bit surprised
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-806296188
@gedOHub Is there anything we can try to reproduce this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mrshu commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
mrshu commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-769433134
@gedOHub We ran into a similar issue when trying to switch to Airflow 2.0.
Our setup seems to be similar to yours -- we also run Airflow in a Docker container. We found out that the problem was that in the `1.x` version that we ran previously we've executed `airflow initdb` on each startup of the Docker container in order to apply the migrations (i.e. to simplify switching to a new version of Airflow).
It seems that in Airflow 2.0 the appropriate command would be `airflow db upgrade` as opposed to `airflow db init`, which ought to really be used just once. Swapping these two commands has fixed the issue for us.
It's a long shot and may not really be the cause of the issue here but I am posting this in here so that others can potentially benefit from it when looking up a similar error message.
cc @Lakeron
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-768377751
@kaxil Is it possible that this issue is related to the [9148](https://github.com/apache/airflow/issues/9148) issue?
I wouldn't expect it, but symptoms seems similar at first glance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-768251036
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13925: Duplicate entry with MySQL backend (AriFlow 2.0.0)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13925:
URL: https://github.com/apache/airflow/issues/13925#issuecomment-914716838
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org