You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/15 01:20:29 UTC
[GitHub] [airflow] doowhtron opened a new issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
doowhtron opened a new issue #13685:
URL: https://github.com/apache/airflow/issues/13685
**Apache Airflow version**: 2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: tencent cloud
- **OS** (e.g. from /etc/os-release): centos7
- **Kernel** (e.g. `uname -a`): 3.10
- **Install tools**:
- **Others**: Server version: 8.0.22 MySQL Community Server - GPL
**What happened**:
Scheduler died when I try to modify a dag's schedule_interval from "None" to "* */1 * * *"(I edited the dag file in the dag folder and saved it). A few minutes later I tried to start the scheduler again and it began to run.
And the logs are as follows:
```
{2021-01-15 09:06:22,636} {{scheduler_job.py:1293}} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
cursor.execute(statement, parameters)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.IntegrityError: (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
self._run_scheduler_loop()
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1474, in _do_scheduling
self._create_dag_runs(query.all(), session)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1561, in _create_dag_runs
dag.create_dagrun(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/utils/session.py", line 62, in wrapper
return func(*args, **kwargs)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/models/dag.py", line 1807, in create_dagrun
session.flush()
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
_emit_insert_statements(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
result = cached_connections[connection].execute(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
cursor.execute(statement, parameters)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/MySQLdb/connections.py", line 239, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")
[SQL: INSERT INTO dag_run (dag_id, execution_date, start_date, end_date, state, run_id, creating_job_id, external_trigger, run_type, conf, last_scheduling_decision, dag_hash) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)]
[parameters: ('huge_demo13499411352', datetime.datetime(2021, 1, 15, 1, 4), datetime.datetime(2021, 1, 15, 1, 6, 22, 629433), None, 'running', 'scheduled__2021-01-15T01:04:00+00:00', 71466, 0, <DagRunType.SCHEDULED: 'scheduled'>, b'\x80\x05}\x94.', None, '60078c379cdeecb9bc8844eed5aa9745')]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
{2021-01-15 09:06:23,648} {{process_utils.py:95}} INFO - Sending Signals.SIGTERM to GPID 66351
{2021-01-15 09:06:23,781} {{process_utils.py:61}} INFO - Process psutil.Process(pid=66351, status='terminated') (66351) terminated with exit code 0
{2021-01-15 09:06:23,781} {{scheduler_job.py:1296}} INFO - Exited execute loop
```
**What you expected to happen**:
Schdeduler should not die.
**How to reproduce it**:
I don't know how to reproduce it
**Anything else we need to know**:
No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767885359
https://github.com/apache/airflow/pull/13920 should fix it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767719293
> I experienced the same issue today while upgrading Airflow from 1.10.14 -> 2.0. Upgrading `mysqlclient` from 1.3.x to 1.4.x didn't seem to help.
>
> By manually deleting the rows in the dagrun table in question we were eventually able to get the scheduler to run reliably, however I would love to know more about what caused these issues (Also, dropping production data isn't a great solution in most cases ๐ )
Can you please provide the steps to reproduce it please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
SamWheating commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767161126
I experienced the same issue today while upgrading Airflow from 1.10.14 -> 2.0. Upgrading `mysqlclient` from 1.3.x to 1.4.x didn't seem to help.
By manually deleting the rows in the dagrun table in question we were eventually able to get the scheduler to run reliably, however I would love to know more about what caused these issues (Also, dropping production data isn't a great solution in most cases ๐ )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] madison-ookla commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
madison-ookla commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-761068131
@doowhtron what version of `mysqlclient` are you running? We found out that we somehow fell back to a 1.3.x version in our new environment and pinning that requirement to >1.4 resolved this issue for us!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] doowhtron commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
doowhtron commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-761912980
> version
mysqlclient 1.4.6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
SamWheating commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767844852
I haven't been able to reproduce this issue outside of our production environment, but after a little more investigation I think that our issue is related to https://github.com/apache/airflow/pull/13278#issuecomment-755218560.
Here's my understanding thus far:
- We were running mysqlclient 1.3.x which I believe led to incorrect values in the `DAG.run_type` field.
- this led to calculating the wrong dates for `next_dagrun` and `next_dagrun_create_after` as the value of most_recent_dag_runs was incorrect.
- This caused the scheduler to recreate old runs which then led to Duplicate Entry errors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatthewRBruce commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
MatthewRBruce commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767719815
Sam and I have been looking at this, and so far hereโs my theory:
It appears this happens when the Scheduler is restarted/killed during an active DagRun. Based on looking at the scheduling code, and the logs, If a DagRun is at itโs max_active_runs (indicated by a log line like this: {scheduler_job.py:1598} INFO - DAG airflow-utils.send-airflow-heartbeat is at (or above) max_active_runs (1 of 1), not creating any more runs ) then the next_dagrun_create_after field for that DagRun will be set to None/NULL here:
https://github.com/apache/airflow/blob/2.0.0/airflow/jobs/scheduler_job.py#L1604
Now if the scheduler is restarted/killed before this Dag Finishes and it starts back up, the DagModel will be returned by `DagModel.dags_needing_dagrun(session)` here:
https://github.com/apache/airflow/blob/2.0.0/airflow/jobs/scheduler_job.py#L1473-L1474
and then a new DagRun will be created for it, resulting in a violation of the Unique Key Constraint
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13685:
URL: https://github.com/apache/airflow/issues/13685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on issue #13685: scheduler dies with "sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'huge_demo13499411352-2021-01-15 01:04:00.000000' for key 'dag_run.dag_id'")"
Posted by GitBox <gi...@apache.org>.
SamWheating commented on issue #13685:
URL: https://github.com/apache/airflow/issues/13685#issuecomment-767161126
I experienced the same issue today while upgrading Airflow from 1.10.14 -> 2.0. Upgrading `mysqlclient` from 1.3.x to 1.4.x didn't seem to help.
By manually deleting the rows in the dagrun table in question we were eventually able to get the scheduler to run reliably, however I would love to know more about what caused these issues (Also, dropping production data isn't a great solution in most cases ๐ )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org