You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/04 10:08:33 UTC
[GitHub] [airflow] ozw1z5rd opened a new issue #10154: DAG_TAG table and foreign key constraint
ozw1z5rd opened a new issue #10154:
URL: https://github.com/apache/airflow/issues/10154
Hello all,
after installing AIRFLOW 1.10.10 I got the dag_tag table defined as
```
CREATE TABLE `dag_tag` (
`name` varchar(100) NOT NULL,
`dag_id` varchar(250) NOT NULL,
PRIMARY KEY (`name`,`dag_id`),
KEY `dag_id` (`dag_id`),
CONSTRAINT `dag_tag_ibfk_1` FOREIGN KEY (`dag_id`) REFERENCES `dag` (`dag_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin;
```
Because of this definition WEB UI, when I delete a dag, displays an exception about the foreign key constraint.
I wonder if this other definition is not a better solution:
```
CREATE TABLE `dag_tag` (
`name` varchar(100) NOT NULL,
`dag_id` varchar(250) NOT NULL,
PRIMARY KEY (`name`,`dag_id`),
KEY `dag_id` (`dag_id`),
CONSTRAINT `dag_tag_ibfk_1` FOREIGN KEY (`dag_id`) REFERENCES `dag` (`dag_id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin;
```
After I changed the table in this way, anything works fine.
I'm using airflow 1.10.10 after migration from 1.9.0
Mysql is 5.7.x
Python 2.7.x
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ozw1z5rd edited a comment on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ozw1z5rd edited a comment on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-674729110
Hello Kaxil, I got the issue when migrating from airflow 1.9.0 to 1.10.10. If I add some dags with tags, when I delete them from the UI, I got exception. Mumble...
`for model in models....`
perhaps is the list of models that comes from my installation different from yours? So that, on my installation it attempts to delete the DAG starting from the wrong table.
What is supposed to return this code? ` models.base.Base._decl_class_registry.values()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-681154077
@ozw1z5rd It will return all the Classes/Models that are used in the project
```
In [1]: from airflow import models
In [3]: list(models.base.Base._decl_class_registry.values())
Out[3]:
[airflow.models.errors.ImportError,
<sqlalchemy.ext.declarative.clsregistry._ModuleMarker at 0x7fc660519ad0>,
airflow.models.dagcode.DagCode,
airflow.models.dagpickle.DagPickle,
airflow.models.dagrun.DagRun,
airflow.models.log.Log,
airflow.models.pool.Pool,
airflow.models.taskfail.TaskFail,
airflow.models.taskreschedule.TaskReschedule,
airflow.models.variable.Variable,
airflow.models.xcom.BaseXCom,
airflow.models.taskinstance.TaskInstance,
airflow.models.dag.DagTag,
airflow.models.dag.DagModel,
airflow.models.connection.Connection,
airflow.models.renderedtifields.RenderedTaskInstanceFields,
airflow.models.slamiss.SlaMiss,
airflow.models.kubernetes.KubeResourceVersion,
airflow.models.kubernetes.KubeWorkerIdentifier,
airflow.models.knownevent.KnownEventType,
airflow.models.knownevent.KnownEvent,
airflow.models.user.User,
airflow.models.chart.Chart]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-810357675
> it ran in staging and later all the differences have been moved by DBA teams on the production db.
If you did not run upgradedb in production you then I'm afraid that all bets are off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ozw1z5rd commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ozw1z5rd commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-682380876
@kaxil on my production installation, the list has different order.
```
<class 'airflow.models.dag.DagModel'>,
<class 'airflow.models.dag.DagTag'>,
```
in your list:
```
<class 'airflow.models.dag.DagTag'>,
<class 'airflow.models.dag.DagModel'>,
```
I did not dive in deep, however I think this problem raises _because of the upgrade_ from airflow 1.9.0 to 1.10.10. The upgradedb did not run in production, it ran in staging and later all the differences have been moved by DBA teams on the production db.
```
>>> pprint.pprint(list(models.base.Base._decl_class_registry.values()))
[<class 'airflow.models.dagrun.DagRun'>,
<class 'airflow.models.xcom.XCom'>,
<class 'airflow.models.log.Log'>,
<class 'airflow.models.kubernetes.KubeResourceVersion'>,
<class 'airflow.models.taskfail.TaskFail'>,
<class 'airflow.models.kubernetes.KubeWorkerIdentifier'>,
<class 'airflow.jobs.base_job.BaseJob'>,
<class 'airflow.models.knownevent.KnownEvent'>,
<class 'airflow.jobs.backfill_job.BackfillJob'>,
<class 'airflow.models.taskinstance.TaskInstance'>,
<class 'airflow.models.chart.Chart'>,
<class 'airflow.models.errors.ImportError'>,
<class 'airflow.models.dagpickle.DagPickle'>,
<class 'airflow.models.user.User'>,
<class 'airflow.models.dag.DagModel'>,
<class 'airflow.models.variable.Variable'>,
<class 'airflow.models.pool.Pool'>,
<class 'airflow.models.dagcode.DagCode'>,
<class 'airflow.models.slamiss.SlaMiss'>,
<class 'airflow.models.connection.Connection'>,
<class 'airflow.jobs.local_task_job.LocalTaskJob'>,
<class 'airflow.models.dag.DagTag'>,
<sqlalchemy.ext.declarative.clsregistry._ModuleMarker object at 0x29b96b0>,
<class 'airflow.models.knownevent.KnownEventType'>,
<class 'airflow.models.renderedtifields.RenderedTaskInstanceFields'>,
<class 'airflow.jobs.scheduler_job.SchedulerJob'>,
<class 'airflow.models.taskreschedule.TaskReschedule'>]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ozw1z5rd commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ozw1z5rd commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-674729110
Hello Kaxil, I got the issue when migrating from airflow 1.9.0 to 1.10.10. If I add some dags with tags, when I delete them from the UI, I got exception. Mumble...
`for model in models....`
perhaps is the list of model that come from my installation different from the one you are getting? So that, on my installation it attempts to delete the DAG starting from the wrong table.
what is supposed to return this code?
` models.base.Base._decl_class_registry.values()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-668508058
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ozw1z5rd commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ozw1z5rd commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-668567527
Ok, I'm at work now. I'll do asap if meanwhile nobody did it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-670940986
I wasn't able to reproduce the error from the Webserver, can you let me know the steps to reproduce it.
I get what you are saying for the SQL statement but the delete_dag deletes all the tables with `dag_id` column:
https://github.com/apache/airflow/blob/215c16f23b4a4ee652e8fe652f1c8b78a4a40aa8/airflow/api/common/experimental/delete_dag.py#L57-L62
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-668545367
Can you open PR? This makes it easier to discuss, verify the change, and close the ticket. At first glance, it looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-681154077
@ozw1z5rd It will return all the Classes/Models that are used in the project
```python
In [1]: from airflow import models
In [3]: list(models.base.Base._decl_class_registry.values())
Out[3]:
[airflow.models.errors.ImportError,
<sqlalchemy.ext.declarative.clsregistry._ModuleMarker at 0x7fc660519ad0>,
airflow.models.dagcode.DagCode,
airflow.models.dagpickle.DagPickle,
airflow.models.dagrun.DagRun,
airflow.models.log.Log,
airflow.models.pool.Pool,
airflow.models.taskfail.TaskFail,
airflow.models.taskreschedule.TaskReschedule,
airflow.models.variable.Variable,
airflow.models.xcom.BaseXCom,
airflow.models.taskinstance.TaskInstance,
airflow.models.dag.DagTag,
airflow.models.dag.DagModel,
airflow.models.connection.Connection,
airflow.models.renderedtifields.RenderedTaskInstanceFields,
airflow.models.slamiss.SlaMiss,
airflow.models.kubernetes.KubeResourceVersion,
airflow.models.kubernetes.KubeWorkerIdentifier,
airflow.models.knownevent.KnownEventType,
airflow.models.knownevent.KnownEvent,
airflow.models.user.User,
airflow.models.chart.Chart]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
ashb closed issue #10154:
URL: https://github.com/apache/airflow/issues/10154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-681154077
@ozw1z5rd It will return all the Classes/Models that are used in the project
```pycon
In [1]: from airflow import models
In [3]: list(models.base.Base._decl_class_registry.values())
Out[3]:
[airflow.models.errors.ImportError,
<sqlalchemy.ext.declarative.clsregistry._ModuleMarker at 0x7fc660519ad0>,
airflow.models.dagcode.DagCode,
airflow.models.dagpickle.DagPickle,
airflow.models.dagrun.DagRun,
airflow.models.log.Log,
airflow.models.pool.Pool,
airflow.models.taskfail.TaskFail,
airflow.models.taskreschedule.TaskReschedule,
airflow.models.variable.Variable,
airflow.models.xcom.BaseXCom,
airflow.models.taskinstance.TaskInstance,
airflow.models.dag.DagTag,
airflow.models.dag.DagModel,
airflow.models.connection.Connection,
airflow.models.renderedtifields.RenderedTaskInstanceFields,
airflow.models.slamiss.SlaMiss,
airflow.models.kubernetes.KubeResourceVersion,
airflow.models.kubernetes.KubeWorkerIdentifier,
airflow.models.knownevent.KnownEventType,
airflow.models.knownevent.KnownEvent,
airflow.models.user.User,
airflow.models.chart.Chart]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10154: DAG_TAG table and foreign key constraint
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10154:
URL: https://github.com/apache/airflow/issues/10154#issuecomment-668545367
Can you open PR? This makes it easier to discuss and verify the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org