You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/25 19:09:37 UTC
[GitHub] [airflow] bbovenzi opened a new pull request, #25961: Add triggered dag runs to dataset events
bbovenzi opened a new pull request, #25961:
URL: https://github.com/apache/airflow/pull/25961
With `created_dagruns` as part of the `DatasetEventModel`, we should include that information in the API & UI.
This adds this to the REST API response. And then displays it on a dataset details page by adding another column showing the count of dag runs created by that dataset event. Clicking on the count will show a modal with a table linking out to each dag run.
<img width="1668" alt="Screen Shot 2022-08-25 at 3 09 13 PM" src="https://user-images.githubusercontent.com/4600967/186748513-e603a221-6ac7-4cda-b879-4c5bf43dbbe5.png">
<img width="891" alt="Screen Shot 2022-08-25 at 3 09 28 PM" src="https://user-images.githubusercontent.com/4600967/186748567-49a23ba6-968a-49f4-bab2-9339118f8349.png">
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25961: Add triggered dag runs to dataset events
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25961:
URL: https://github.com/apache/airflow/pull/25961#discussion_r958744801
##########
airflow/www/static/js/types/api-generated.ts:
##########
@@ -1536,9 +1536,54 @@ export interface components {
source_run_id?: string | null;
/** @description The task map index that updated the dataset. */
source_map_index?: number | null;
+ created_dagruns?: components["schemas"]["BasicDAGRun"][];
/** @description The dataset event creation time */
timestamp?: string;
};
+ BasicDAGRun: {
+ /** @description Run ID. */
+ run_id?: string | null;
+ dag_id?: string;
+ /**
+ * Format: date-time
+ * @description The logical date (previously called execution date). This is the time or interval covered by
+ * this DAG run, according to the DAG definition.
+ *
+ * The value of this field can be set only when creating the object. If you try to modify the
+ * field of an existing object, the request fails with an BAD_REQUEST error.
+ *
+ * This together with DAG_ID are a unique key.
+ *
+ * *New in version 2.2.0*
+ */
+ logical_date?: string | null;
Review Comment:
This is just following the regular DagRun schema which has all of these as nullable in `v1.yaml`. Should we update that too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi merged pull request #25961: Add triggered dag runs to dataset events
Posted by GitBox <gi...@apache.org>.
bbovenzi merged PR #25961:
URL: https://github.com/apache/airflow/pull/25961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #25961: Add triggered dag runs to dataset events
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #25961:
URL: https://github.com/apache/airflow/pull/25961#discussion_r958661820
##########
airflow/www/static/js/types/api-generated.ts:
##########
@@ -1536,9 +1536,54 @@ export interface components {
source_run_id?: string | null;
/** @description The task map index that updated the dataset. */
source_map_index?: number | null;
+ created_dagruns?: components["schemas"]["BasicDAGRun"][];
/** @description The dataset event creation time */
timestamp?: string;
};
+ BasicDAGRun: {
+ /** @description Run ID. */
+ run_id?: string | null;
+ dag_id?: string;
+ /**
+ * Format: date-time
+ * @description The logical date (previously called execution date). This is the time or interval covered by
+ * this DAG run, according to the DAG definition.
+ *
+ * The value of this field can be set only when creating the object. If you try to modify the
+ * field of an existing object, the request fails with an BAD_REQUEST error.
+ *
+ * This together with DAG_ID are a unique key.
+ *
+ * *New in version 2.2.0*
+ */
+ logical_date?: string | null;
+ /**
+ * Format: date-time
+ * @deprecated
+ * @description The execution date. This is the same as logical_date, kept for backwards compatibility.
+ * If both this field and logical_date are provided but with different values, the request
+ * will fail with an BAD_REQUEST error.
+ *
+ * *Changed in version 2.2.0*: Field becomes nullable.
+ *
+ * *Deprecated since version 2.2.0*: Use 'logical_date' instead.
+ */
+ execution_date?: string | null;
Review Comment:
Where's this coming from? I don't see it in the schema for BasicDag Run
##########
airflow/www/static/js/types/api-generated.ts:
##########
@@ -1536,9 +1536,54 @@ export interface components {
source_run_id?: string | null;
/** @description The task map index that updated the dataset. */
source_map_index?: number | null;
+ created_dagruns?: components["schemas"]["BasicDAGRun"][];
/** @description The dataset event creation time */
timestamp?: string;
};
+ BasicDAGRun: {
+ /** @description Run ID. */
+ run_id?: string | null;
+ dag_id?: string;
+ /**
+ * Format: date-time
+ * @description The logical date (previously called execution date). This is the time or interval covered by
+ * this DAG run, according to the DAG definition.
+ *
+ * The value of this field can be set only when creating the object. If you try to modify the
+ * field of an existing object, the request fails with an BAD_REQUEST error.
+ *
+ * This together with DAG_ID are a unique key.
+ *
+ * *New in version 2.2.0*
+ */
+ logical_date?: string | null;
Review Comment:
None of these should be nullable.
##########
airflow/api_connexion/schemas/dataset_schema.py:
##########
@@ -91,6 +92,25 @@ class DatasetCollectionSchema(Schema):
dataset_collection_schema = DatasetCollectionSchema()
+class BasicDAGRunSchema(SQLAlchemySchema):
+ """Basic Schema for DAGRun"""
+
+ class Meta:
+ """Meta"""
+
+ model = DagRun
+ dateformat = "iso"
+
+ run_id = auto_field(data_key='dag_run_id')
+ dag_id = auto_field(dump_only=True)
+ execution_date = auto_field(data_key="logical_date", dump_only=True)
Review Comment:
execution_date is only included for compat reasons, let's not include it in a new API.
```suggestion
logical_date = auto_field(dump_only=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25961: Add triggered dag runs to dataset events
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25961:
URL: https://github.com/apache/airflow/pull/25961#discussion_r958745151
##########
airflow/www/static/js/types/api-generated.ts:
##########
@@ -1536,9 +1536,54 @@ export interface components {
source_run_id?: string | null;
/** @description The task map index that updated the dataset. */
source_map_index?: number | null;
+ created_dagruns?: components["schemas"]["BasicDAGRun"][];
/** @description The dataset event creation time */
timestamp?: string;
};
+ BasicDAGRun: {
+ /** @description Run ID. */
+ run_id?: string | null;
+ dag_id?: string;
+ /**
+ * Format: date-time
+ * @description The logical date (previously called execution date). This is the time or interval covered by
+ * this DAG run, according to the DAG definition.
+ *
+ * The value of this field can be set only when creating the object. If you try to modify the
+ * field of an existing object, the request fails with an BAD_REQUEST error.
+ *
+ * This together with DAG_ID are a unique key.
+ *
+ * *New in version 2.2.0*
+ */
+ logical_date?: string | null;
+ /**
+ * Format: date-time
+ * @deprecated
+ * @description The execution date. This is the same as logical_date, kept for backwards compatibility.
+ * If both this field and logical_date are provided but with different values, the request
+ * will fail with an BAD_REQUEST error.
+ *
+ * *Changed in version 2.2.0*: Field becomes nullable.
+ *
+ * *Deprecated since version 2.2.0*: Use 'logical_date' instead.
+ */
+ execution_date?: string | null;
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25961: Add triggered dag runs to dataset events
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25961:
URL: https://github.com/apache/airflow/pull/25961#discussion_r958746181
##########
airflow/api_connexion/schemas/dataset_schema.py:
##########
@@ -91,6 +92,25 @@ class DatasetCollectionSchema(Schema):
dataset_collection_schema = DatasetCollectionSchema()
+class BasicDAGRunSchema(SQLAlchemySchema):
+ """Basic Schema for DAGRun"""
+
+ class Meta:
+ """Meta"""
+
+ model = DagRun
+ dateformat = "iso"
+
+ run_id = auto_field(data_key='dag_run_id')
+ dag_id = auto_field(dump_only=True)
+ execution_date = auto_field(data_key="logical_date", dump_only=True)
Review Comment:
the dag run model still uses `execution_date` but the `data_key`, changes the name to logical date for the API docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org