You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/09 10:10:24 UTC
[GitHub] [airflow] turbaszek opened a new pull request #8227: Add run_type
to DagRun
turbaszek opened a new pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227
https://issues.apache.org/jira/browse/AIRFLOW-6242
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406111488
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
Now I'm wondering, should we set the run_type using run_id? In example by checking the prefix. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406106007
##########
File path: airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py
##########
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Add DagRun run_type
+
+Revision ID: 3c20cacc0044
+Revises: 952da73b5eff
+Create Date: 2020-04-08 13:35:25.671327
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.models import DagRun
Review comment:
Please look at:airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py line 47
```
class TaskInstance(Base):
__tablename__ = "task_instance"
task_id = Column(String(ID_LEN, **COLLATION_ARGS), primary_key=True)
dag_id = Column(String(ID_LEN, **COLLATION_ARGS), primary_key=True)
execution_date = Column(sa.DateTime, primary_key=True)
max_tries = Column(Integer)
try_number = Column(Integer, default=0)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406111618
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
If run_id is not passed, it will be generated based on execution_date + run_type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406108915
##########
File path: airflow/models/dagrun.py
##########
@@ -65,14 +66,15 @@ class DagRun(Base, LoggingMixin):
)
def __init__(self, dag_id=None, run_id=None, execution_date=None, start_date=None, external_trigger=None,
- conf=None, state=None):
+ conf=None, state=None, run_type=None):
self.dag_id = dag_id
self.run_id = run_id
self.execution_date = execution_date
self.start_date = start_date
self.external_trigger = external_trigger
self.conf = conf
self.state = state
+ self.run_type = run_type
Review comment:
Yeah, we should look at how we query dag_run table, as I'd guess we want a multi-column index on run_type + something else (exeuction date? dag id?) as we don't (I think) ever select just by run_type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406106635
##########
File path: airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py
##########
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Add DagRun run_type
+
+Revision ID: 3c20cacc0044
+Revises: 952da73b5eff
+Create Date: 2020-04-08 13:35:25.671327
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.models import DagRun
Review comment:
Yup, see #8176 for an example of how to do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406107829
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
@ashb you can provide `run_id` or `run_type + execution_date`. If you don't provide `run_id` it will be autogenerated. In most places, the `run_id` was created in the following way: `{run_type}__{execution_date.isoformat()}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406107134
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
Oh I see, defaulted/handled in create_dagrun, nice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8227: Add run_type to
DagRun
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#issuecomment-615178713
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=h1) Report
> Merging [#8227](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/79d3f33c1b65c9c7e7b1a75e25d38cab9aa4517f&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `14.63%`.
[](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8227 +/- ##
======================================
Coverage 6.23% 6.24%
======================================
Files 941 941
Lines 45644 45667 +23
======================================
+ Hits 2846 2851 +5
- Misses 42798 42816 +18
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `0.00% <ø> (ø)` | |
| [airflow/api/common/experimental/trigger\_dag.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC90cmlnZ2VyX2RhZy5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `0.00% <ø> (ø)` | |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `0.00% <0.00%> (ø)` | |
| [airflow/operators/subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3ViZGFnX29wZXJhdG9yLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/ti\_deps/deps/dagrun\_id\_dep.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZGFncnVuX2lkX2RlcC5weQ==) | `69.23% <0.00%> (-2.20%)` | :arrow_down: |
| [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `0.00% <0.00%> (ø)` | |
| [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `26.47% <11.11%> (-0.18%)` | :arrow_down: |
| [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `27.19% <33.33%> (+0.47%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/airflow/pull/8227/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=footer). Last update [79d3f33...a5b36b7](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406108194
##########
File path: airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py
##########
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Add DagRun run_type
+
+Revision ID: 3c20cacc0044
+Revises: 952da73b5eff
+Create Date: 2020-04-08 13:35:25.671327
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.models import DagRun
+from airflow.utils.types import DagRunType
+
+# revision identifiers, used by Alembic.
+revision = "3c20cacc0044"
+down_revision = "952da73b5eff"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ """Apply Add DagRun run_type"""
+ op.add_column("dag_run", sa.Column("run_type", sa.String(length=50), nullable=True))
+
+ connection = op.get_bind()
+ sessionmaker = sa.orm.sessionmaker()
+ session = sessionmaker(bind=connection)
+
+ for run_type in DagRunType:
+ session.query(DagRun).filter(DagRun.run_id.like(f"{run_type.value}__%")).update(
+ {DagRun.run_type: run_type.value}, synchronize_session=False
+ )
Review comment:
Run_id can be anything in case of a manual run, so I think we also need a `SET run_type = 'manual' where run_type is null` afterwards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406102543
##########
File path: airflow/models/dagrun.py
##########
@@ -65,14 +66,15 @@ class DagRun(Base, LoggingMixin):
)
def __init__(self, dag_id=None, run_id=None, execution_date=None, start_date=None, external_trigger=None,
- conf=None, state=None):
+ conf=None, state=None, run_type=None):
self.dag_id = dag_id
self.run_id = run_id
self.execution_date = execution_date
self.start_date = start_date
self.external_trigger = external_trigger
self.conf = conf
self.state = state
+ self.run_type = run_type
Review comment:
Probably it's worth to use this column in index as mentioned in the original issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406111618
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
If run_id is empty, it will be generated based on execution_date + run_type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406104850
##########
File path: airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py
##########
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Add DagRun run_type
+
+Revision ID: 3c20cacc0044
+Revises: 952da73b5eff
+Create Date: 2020-04-08 13:35:25.671327
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.models import DagRun
Review comment:
I'm afraid we can't use this class in migration. This class may change in the future, which will damage our old migrations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406106150
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
What happend to run_id? I think we still need this, even if we don't use the prefix for detecting the type
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #8227: Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#issuecomment-611464657
Oh and we should add the run_type column to the DAGRunModelView.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406112369
##########
File path: airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py
##########
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Add DagRun run_type
+
+Revision ID: 3c20cacc0044
+Revises: 952da73b5eff
+Create Date: 2020-04-08 13:35:25.671327
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.models import DagRun
Review comment:
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8227: Add
run_type to DagRun
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406122863
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
I think run_type should be non-null in the db, yes. So perhaps make run_type required, but run_id can be overridden.
We could detect run_type based on run_id, it would make it easier for anyone to upgrade, sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406112990
##########
File path: airflow/api/common/experimental/mark_tasks.py
##########
@@ -48,11 +48,11 @@ def _create_dagruns(dag, execution_dates, state, run_type):
for date in dates_to_create:
dag_run = dag.create_dagrun(
- run_id=f"{run_type}__{date.isoformat()}",
Review comment:
Obtaining run_type from run_id may help because then we will always have run_type thus it can be used more effectively in index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8227:
Add run_type to DagRun
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8227: Add run_type to DagRun
URL: https://github.com/apache/airflow/pull/8227#discussion_r406102543
##########
File path: airflow/models/dagrun.py
##########
@@ -65,14 +66,15 @@ class DagRun(Base, LoggingMixin):
)
def __init__(self, dag_id=None, run_id=None, execution_date=None, start_date=None, external_trigger=None,
- conf=None, state=None):
+ conf=None, state=None, run_type=None):
self.dag_id = dag_id
self.run_id = run_id
self.execution_date = execution_date
self.start_date = start_date
self.external_trigger = external_trigger
self.conf = conf
self.state = state
+ self.run_type = run_type
Review comment:
Probably it's worth to use this column in index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services