You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/23 19:46:59 UTC
[GitHub] [airflow] potiuk opened a new pull request #10501: Make DagRun and Dag pylint-compatible
potiuk opened a new pull request #10501:
URL: https://github.com/apache/airflow/pull/10501
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#discussion_r475720645
##########
File path: airflow/models/dag.py
##########
@@ -1469,7 +1515,7 @@ def create_dagrun(self,
run_id=None,
start_date=None,
external_trigger=False,
- conf=None,
+ conf_dict=None,
Review comment:
True. Will go back with that - besides seems some cyclic imports need to be solve as well. I ill let the other "pylint" changes to be merged and will take a stab (I think one last time :) ) into this one ... We are basically 3 files to go :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679403362
It will likely fail with cyclic impots we will have to resolve though :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #10501:
URL: https://github.com/apache/airflow/pull/10501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679022576
I'm afraid that this will require more changes due to cyclic imports :<
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#discussion_r475710256
##########
File path: airflow/models/dag.py
##########
@@ -1469,7 +1515,7 @@ def create_dagrun(self,
run_id=None,
start_date=None,
external_trigger=False,
- conf=None,
+ conf_dict=None,
Review comment:
This is a breaking change, many users use keyword args
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679406702
Tomorrow :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679405606
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.models.dag -> airflow.operators.subdag_operator) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.models -> airflow.models.dag -> airflow.sensors.external_task_sensor) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.models -> airflow.models.dagbag -> airflow.models.dag -> airflow.sensors.external_task_sensor -> airflow.operators.dummy_operator) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.api.common.experimental.get_task_instance -> airflow.models -> airflow.models.dag -> airflow.operators.subdag_operator) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.models -> airflow.models.dagbag -> airflow.models.serialized_dag -> airflow.serialization.serialized_objects -> airflow.models.dag -> airflow.sensors.external_task_sensor -> airflow.operators.dummy_operator) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.models -> airflow.models.dagbag -> airflow.models.serialized_dag -> airflow.models.dag -> airflow.sensors.external_task_sensor -> airflow.operators.dummy_operator) (cyclic-import)
airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py:1:0: R0401: Cyclic import (airflow.api.common.experimental -> airflow.models -> airflow.models.dag -> airflow.operators.subdag_operator -> airflow.api.common.experimental.get_task_instance) (cyclic-import)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#discussion_r475846873
##########
File path: airflow/models/dag.py
##########
@@ -1469,7 +1515,7 @@ def create_dagrun(self,
run_id=None,
start_date=None,
external_trigger=False,
- conf=None,
+ conf_dict=None,
Review comment:
Resolved.
##########
File path: airflow/models/dag.py
##########
@@ -1865,22 +1925,31 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: List[str], session=None):
dag_models = session.query(cls).all()
try:
- for dag_model in dag_models:
- if dag_model.fileloc is not None:
- if correct_maybe_zipped(dag_model.fileloc) not in alive_dag_filelocs:
- dag_model.is_active = False
- else:
- # If is_active is set as False and the DAG File still exists
- # Change is_active=True
- if not dag_model.is_active:
- dag_model.is_active = True
- else:
- continue
+ cls._deactivate_deleted_dags_in_models(alive_dag_filelocs, dag_models)
session.commit()
except Exception:
session.rollback()
raise
+ @classmethod
+ def _deactivate_deleted_dags_in_models(cls, alive_dag_filelocs, dag_models):
+ """Loop through all models and deactivate te DAGs."""
Review comment:
Resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-706789502
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#discussion_r475535175
##########
File path: airflow/models/dag.py
##########
@@ -1865,22 +1925,31 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: List[str], session=None):
dag_models = session.query(cls).all()
try:
- for dag_model in dag_models:
- if dag_model.fileloc is not None:
- if correct_maybe_zipped(dag_model.fileloc) not in alive_dag_filelocs:
- dag_model.is_active = False
- else:
- # If is_active is set as False and the DAG File still exists
- # Change is_active=True
- if not dag_model.is_active:
- dag_model.is_active = True
- else:
- continue
+ cls._deactivate_deleted_dags_in_models(alive_dag_filelocs, dag_models)
session.commit()
except Exception:
session.rollback()
raise
+ @classmethod
+ def _deactivate_deleted_dags_in_models(cls, alive_dag_filelocs, dag_models):
+ """Loop through all models and deactivate te DAGs."""
Review comment:
```suggestion
"""Loop through all models and deactivate DAGs."""
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679335407
I want to rebase that one on top of #10499 to see if the cyclic dependencies problem will be fixed,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10501: [WIP] Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#issuecomment-679402567
This is the last one @BasPH ...... https://lists.apache.org/thread.html/f4940d36e98ded96a2473bb2ccdfa4cc648faa2c1334b2aa901c0bba%40%3Cdev.airflow.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10501: Make DagRun and Dag pylint-compatible
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10501:
URL: https://github.com/apache/airflow/pull/10501#discussion_r475720645
##########
File path: airflow/models/dag.py
##########
@@ -1469,7 +1515,7 @@ def create_dagrun(self,
run_id=None,
start_date=None,
external_trigger=False,
- conf=None,
+ conf_dict=None,
Review comment:
True. Will go back with that - besides seems some cyclic imports need to be solved as well. I ill let the other "pylint" changes to be merged and will take a stab (I think one last time :) ) into this one ... We are basically 3 files to go :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org