You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/22 12:03:42 UTC
[GitHub] [airflow] buxizhizhoum opened a new pull request #16584: Modify models/dag.py to let it pass pylint check
buxizhizhoum opened a new pull request #16584:
URL: https://github.com/apache/airflow/pull/16584
Modified the airflow/models/dag.py model to let it pass static code check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-868517028
(and runing docs build locally is also fast now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-868181524
> Some static checks/docs/pylint still fail
@potiuk Sorry for disturbing, I have fixed them, could you please review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657022881
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
It is better than what I did, thanks for the advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869660090
shall we abandon this in the light of #16682 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum closed pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum closed pull request #16584:
URL: https://github.com/apache/airflow/pull/16584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657047364
##########
File path: airflow/models/dag.py
##########
@@ -2122,7 +2165,7 @@ class DagTag(Base):
dag_id = Column(String(ID_LEN), ForeignKey('dag.dag_id'), primary_key=True)
def __repr__(self):
- return self.name
+ return f"<DagTag: {self.name}>"
Review comment:
This change has been rolled back, the original `return self.name` triggers an pylint error(__repr__ does not return str (invalid-repr-returned)), and I change it a little to suppress it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657051603
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
I have placed a DeprecationWarning to this method, and applied some change to suppress pylint error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657015836
##########
File path: airflow/models/dag.py
##########
@@ -267,9 +266,10 @@ def __init__(
jinja_environment_kwargs: Optional[Dict] = None,
render_template_as_native_obj: bool = False,
tags: Optional[List[str]] = None,
- ):
+ ): # pylint: disable=too-many-arguments,too-many-locals,too-many-statements
from airflow.utils.task_group import TaskGroup
+ super().__init__()
Review comment:
This is to suppress `super-init-not-called` error of pylint, it might be better than disable pylint check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869466096
Btw. We are getting rid of puking today (see the devlist) so this change is really obsolete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657017807
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
You might as well be right (there are a lot of dead code in Airflow). But we can’t just remove it due to backward compatibility. Maybe add a DeprecationWarning (to plan for a removal) and just ignore the entire block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum edited a comment on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum edited a comment on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-866511371
@ashb Could you have a look on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657016810
##########
File path: airflow/models/dag.py
##########
@@ -267,9 +266,10 @@ def __init__(
jinja_environment_kwargs: Optional[Dict] = None,
render_template_as_native_obj: bool = False,
tags: Optional[List[str]] = None,
- ):
+ ): # pylint: disable=too-many-arguments,too-many-locals,too-many-statements
from airflow.utils.task_group import TaskGroup
+ super().__init__()
Review comment:
IMO this is a OK fix. Thanks for the explaination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657015836
##########
File path: airflow/models/dag.py
##########
@@ -267,9 +266,10 @@ def __init__(
jinja_environment_kwargs: Optional[Dict] = None,
render_template_as_native_obj: bool = False,
tags: Optional[List[str]] = None,
- ):
+ ): # pylint: disable=too-many-arguments,too-many-locals,too-many-statements
from airflow.utils.task_group import TaskGroup
+ super().__init__()
Review comment:
This is to suppress `super-init-not-called` error of pylint, I think it might be better than disable pylint check?
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
I searched the project, it seems that this method is not used already, Or may be I am wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum closed pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum closed pull request #16584:
URL: https://github.com/apache/airflow/pull/16584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869301306
> Still some errors.
>
> I HEARTILY recommend installing pre-commits locally and running them for every commit.
my bad, I will fix as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-867847798
Some static checks/docs/pylint still fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r658044490
##########
File path: airflow/models/dag.py
##########
@@ -694,22 +701,27 @@ def has_dag_runs(self, session=None, include_externally_triggered=True) -> bool:
@property
def dag_id(self) -> str:
+ """Get the dag_id"""
return self._dag_id
@dag_id.setter
def dag_id(self, value: str) -> None:
+ """Set dag_id"""
self._dag_id = value
@property
def full_filepath(self) -> str:
+ """Dag location"""
return self._full_filepath
@full_filepath.setter
def full_filepath(self, value) -> None:
+ """update full filepath"""
self._full_filepath = value
@property
def concurrency(self) -> int:
+ """Max active tasks limit of a dag"""
Review comment:
@ashb Could you help to have a look again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869660090
shall we abandon this in the light of #16682 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869684068
> shall we abandon this in the light of #16682 ?
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-869684068
> shall we abandon this in the light of #16682 ?
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657279141
##########
File path: airflow/models/dag.py
##########
@@ -694,22 +701,27 @@ def has_dag_runs(self, session=None, include_externally_triggered=True) -> bool:
@property
def dag_id(self) -> str:
+ """Get the dag_id"""
return self._dag_id
@dag_id.setter
def dag_id(self, value: str) -> None:
+ """Set dag_id"""
self._dag_id = value
@property
def full_filepath(self) -> str:
+ """Dag location"""
return self._full_filepath
@full_filepath.setter
def full_filepath(self, value) -> None:
+ """update full filepath"""
self._full_filepath = value
@property
def concurrency(self) -> int:
+ """Max active tasks limit of a dag"""
Review comment:
I have removed less helpful doc strings including this one, thanks for your advice:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-866511371
@ashb Could you have a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-866779020
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#issuecomment-868516706
Still some errors.
I HEARTILY recommend installing pre-commits locally and running them for every commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657017807
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
You might as well be right (there are a lot of dead code in Airflow). But we can’t just remove it due to backward compatibility. Maybe add a DeprecationWarning (to plan for a removal) and ignore the entire block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] buxizhizhoum commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
buxizhizhoum commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657016282
##########
File path: airflow/models/dag.py
##########
@@ -2122,7 +2165,7 @@ class DagTag(Base):
dag_id = Column(String(ID_LEN), ForeignKey('dag.dag_id'), primary_key=True)
def __repr__(self):
- return self.name
+ return f"<DagTag: {self.name}>"
Review comment:
Yes, I will rollback this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r656985554
##########
File path: airflow/models/dag.py
##########
@@ -2122,7 +2165,7 @@ class DagTag(Base):
dag_id = Column(String(ID_LEN), ForeignKey('dag.dag_id'), primary_key=True)
def __repr__(self):
- return self.name
+ return f"<DagTag: {self.name}>"
Review comment:
This change doesn’t seem related.
##########
File path: airflow/models/dag.py
##########
@@ -267,9 +266,10 @@ def __init__(
jinja_environment_kwargs: Optional[Dict] = None,
render_template_as_native_obj: bool = False,
tags: Optional[List[str]] = None,
- ):
+ ): # pylint: disable=too-many-arguments,too-many-locals,too-many-statements
from airflow.utils.task_group import TaskGroup
+ super().__init__()
Review comment:
Is this needed?
##########
File path: airflow/models/dag.py
##########
@@ -1564,72 +1600,78 @@ def filter_task_group(group, parent_group):
group.upstream_task_ids = group.upstream_task_ids.intersection(dag.task_dict.keys())
group.downstream_task_ids = group.downstream_task_ids.intersection(dag.task_dict.keys())
- for t in dag.tasks:
+ for task in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# make the cut
- t._upstream_task_ids = t.upstream_task_ids.intersection(dag.task_dict.keys())
- t._downstream_task_ids = t.downstream_task_ids.intersection(dag.task_dict.keys())
+ task._upstream_task_ids = task.upstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
+ task._downstream_task_ids = task.downstream_task_ids.intersection(dag.task_dict.keys()) # pylint: disable=protected-access
if len(dag.tasks) < len(self.tasks):
dag.partial = True
return dag
def has_task(self, task_id: str):
+ """check whether task specified by task_id belongs to this dag"""
return task_id in (t.task_id for t in self.tasks)
def get_task(self, task_id: str, include_subdags: bool = False) -> BaseOperator:
+ """get task from dag"""
if task_id in self.task_dict:
return self.task_dict[task_id]
if include_subdags:
- for dag in self.subdags:
+ for dag in self.subdags: # pylint: disable=redefined-outer-name
if task_id in dag.task_dict:
return dag.task_dict[task_id]
raise TaskNotFound(f"Task {task_id} not found")
- def pickle_info(self):
- d = {}
- d['is_picklable'] = True
- try:
- dttm = timezone.utcnow()
- pickled = pickle.dumps(self)
- d['pickle_len'] = len(pickled)
- d['pickling_duration'] = str(timezone.utcnow() - dttm)
- except Exception as e:
- self.log.debug(e)
- d['is_picklable'] = False
- d['stacktrace'] = traceback.format_exc()
- return d
+ # def pickle_info(self):
+ # d = {}
+ # d['is_picklable'] = True
+ # try:
+ # dttm = timezone.utcnow()
+ # pickled = pickle.dumps(self)
+ # d['pickle_len'] = len(pickled)
+ # d['pickling_duration'] = str(timezone.utcnow() - dttm)
+ # except Exception as e:
+ # self.log.debug(e)
+ # d['is_picklable'] = False
+ # d['stacktrace'] = traceback.format_exc()
+ # return d
Review comment:
Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #16584: Modify models/dag.py to let it pass pylint check
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #16584:
URL: https://github.com/apache/airflow/pull/16584#discussion_r657133978
##########
File path: airflow/models/dag.py
##########
@@ -694,22 +701,27 @@ def has_dag_runs(self, session=None, include_externally_triggered=True) -> bool:
@property
def dag_id(self) -> str:
+ """Get the dag_id"""
return self._dag_id
@dag_id.setter
def dag_id(self, value: str) -> None:
+ """Set dag_id"""
self._dag_id = value
@property
def full_filepath(self) -> str:
+ """Dag location"""
return self._full_filepath
@full_filepath.setter
def full_filepath(self, value) -> None:
+ """update full filepath"""
self._full_filepath = value
@property
def concurrency(self) -> int:
+ """Max active tasks limit of a dag"""
Review comment:
I find these type of DAG comments _less_ helpful than no doc at all.
I'd rather we ignored the warning on these ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org