You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "azharizz (via GitHub)" <gi...@apache.org> on 2025/03/08 05:21:43 UTC
[PR] Update documentation priority weight [airflow]
azharizz opened a new pull request, #47529:
URL: https://github.com/apache/airflow/pull/47529
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
closes: #46824
[link](https://github.com/apache/airflow/issues/46824)
I have already tried the custom weight rule on my local environment, and it is applicable. However, the code block inside it is not the same as the code above. So i try to fix it with add some command airflow plugins to see list plugin created.
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #47529:
URL: https://github.com/apache/airflow/pull/47529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "azharizz (via GitHub)" <gi...@apache.org>.
azharizz commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2709119904
thanks for the review @jscheffl , i have changed the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1986351617
##########
docs/apache-airflow/administration-and-deployment/priority-weight.rst:
##########
@@ -84,21 +84,32 @@ registering it in a plugin.
:end-before: [END custom_priority_weight_strategy]
-Then to use it, you can create an instance of the custom class and provide it in the ``weight_rule`` parameter
-of the task or provide the path of the custom class:
+To check if the custom priority weight strategy is already available in Airflow, you can run the bash command
+``airflow plugins``. Then to use it, you can create an instance of the custom class and provide it in the
+``weight_rule`` parameter of the task or provide the path of the custom class:
.. code-block:: python
- from custom_weight_rule_module import CustomPriorityWeightStrategy
+ from decreasing_priority_weight_strategy import DecreasingPriorityStrategy
# provide the class instance
- task1 = BashOperator(task_id="task", bash_command="echo 1", weight_rule=CustomPriorityWeightStrategy())
+ task1 = BashOperator(task_id="task", bash_command="echo 1", weight_rule=DecreasingPriorityStrategy())
# or provide the path of the class
task1 = BashOperator(
task_id="task",
bash_command="echo 1",
- weight_rule="custom_weight_rule_module.CustomPriorityWeightStrategy",
+ weight_rule="decreasing_priority_weight_strategy.DecreasingPriorityStrategy",
)
Review Comment:
Looking good. Just one nit: You don't need the code-block if the example is included below. Else the code is two times in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1986376919
##########
airflow/example_dags/example_custom_weight.py:
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example DAG demonstrating the usage of the CustomWeightRule."""
+
+from __future__ import annotations
+
+import datetime
+
+import pendulum
+# [START example_custom_weight_dag]
+from airflow.models.dag import DAG
+from airflow.providers.standard.operators.bash import BashOperator
+from airflow.providers.standard.operators.empty import EmptyOperator
+from decreasing_priority_weight_strategy import DecreasingPriorityStrategy
+
+with DAG(
+ dag_id="example_custom_weight",
+ schedule="0 0 * * *",
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ dagrun_timeout=datetime.timedelta(minutes=60),
+ tags=["example", "example2"],
+) as dag:
+
+ start = EmptyOperator(
+ task_id="start",
+ )
+
+ # provide the class instance
+ task_1 = BashOperator(
+ task_id="task_1",
+ bash_command="echo 1",
+ weight_rule=DecreasingPriorityStrategy()
+ )
+
+ #or provide the path of the class
+ task_2 = BashOperator(
+ task_id="task_2",
+ bash_command="echo 1",
+ weight_rule="decreasing_priority_weight_strategy.DecreasingPriorityStrategy"
Review Comment:
Same here:
```suggestion
weight_rule="airflow.example_dags.plugins.decreasing_priority_weight_strategy.DecreasingPriorityStrategy"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "azharizz (via GitHub)" <gi...@apache.org>.
azharizz commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2708682012
sure @jscheffl , i already update the code to add example dags. Also for hte test on the dag and documentation using breeze there is no issue regarding this custom weight rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "azharizz (via GitHub)" <gi...@apache.org>.
azharizz commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1986441764
##########
docs/apache-airflow/administration-and-deployment/priority-weight.rst:
##########
@@ -84,21 +84,32 @@ registering it in a plugin.
:end-before: [END custom_priority_weight_strategy]
-Then to use it, you can create an instance of the custom class and provide it in the ``weight_rule`` parameter
-of the task or provide the path of the custom class:
+To check if the custom priority weight strategy is already available in Airflow, you can run the bash command
+``airflow plugins``. Then to use it, you can create an instance of the custom class and provide it in the
+``weight_rule`` parameter of the task or provide the path of the custom class:
.. code-block:: python
- from custom_weight_rule_module import CustomPriorityWeightStrategy
+ from decreasing_priority_weight_strategy import DecreasingPriorityStrategy
# provide the class instance
- task1 = BashOperator(task_id="task", bash_command="echo 1", weight_rule=CustomPriorityWeightStrategy())
+ task1 = BashOperator(task_id="task", bash_command="echo 1", weight_rule=DecreasingPriorityStrategy())
# or provide the path of the class
task1 = BashOperator(
task_id="task",
bash_command="echo 1",
- weight_rule="custom_weight_rule_module.CustomPriorityWeightStrategy",
+ weight_rule="decreasing_priority_weight_strategy.DecreasingPriorityStrategy",
)
Review Comment:
thank you for checking, i thought it's already removed before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2708186640
For me the change is okay. To be on the safe side... one improvement idea: Can you add the code snippet rather as a new example DAG (e.g. name it `example_custom_weight.py`)? Then you can change the RST to include the code snippet and when loading Airflow with example DAGs you can see how it is working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "ninsbl (via GitHub)" <gi...@apache.org>.
ninsbl commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2711888093
Nice. Thanks, @azharizz !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2711115504
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "enchant3dmango (via GitHub)" <gi...@apache.org>.
enchant3dmango commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1987679017
##########
airflow/example_dags/example_custom_weight.py:
##########
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example DAG demonstrating the usage of the CustomWeightRule."""
+
+from __future__ import annotations
+
+import datetime
+
+import pendulum
+from airflow.example_dags.plugins.decreasing_priority_weight_strategy import DecreasingPriorityStrategy
Review Comment:
I think we need to add a new line after line 24 to avoid [ruff](https://github.com/apache/airflow/blob/main/.pre-commit-config.yaml#L358) static check error.
The error happens in #47406 CI.
The screenshot is attached below.
<img width="667" alt="image" src="https://github.com/user-attachments/assets/34172bc9-2f94-43cc-a8f0-f5eee7ea990a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2708167917
@ninsbl reported on the issue that is possibly a bug rather than just a doc update. Is that still the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1986376872
##########
airflow/example_dags/example_custom_weight.py:
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example DAG demonstrating the usage of the CustomWeightRule."""
+
+from __future__ import annotations
+
+import datetime
+
+import pendulum
+# [START example_custom_weight_dag]
+from airflow.models.dag import DAG
+from airflow.providers.standard.operators.bash import BashOperator
+from airflow.providers.standard.operators.empty import EmptyOperator
+from decreasing_priority_weight_strategy import DecreasingPriorityStrategy
Review Comment:
Just checked, the import directly from plugins is not working, in the examples it is working if you make it an absolute path (as load-able via the airflow package)
```suggestion
from airflow.example_dags.plugins.decreasing_priority_weight_strategy import DecreasingPriorityStrategy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "enchant3dmango (via GitHub)" <gi...@apache.org>.
enchant3dmango commented on code in PR #47529:
URL: https://github.com/apache/airflow/pull/47529#discussion_r1987679017
##########
airflow/example_dags/example_custom_weight.py:
##########
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example DAG demonstrating the usage of the CustomWeightRule."""
+
+from __future__ import annotations
+
+import datetime
+
+import pendulum
+from airflow.example_dags.plugins.decreasing_priority_weight_strategy import DecreasingPriorityStrategy
Review Comment:
I think we need to add a new line after line 24 to avoid [ruff](https://github.com/apache/airflow/blob/main/.pre-commit-config.yaml#L358) static check error.
The error happens in #47406 CI.
The screenshot is attached below.
<img width="667" alt="image" src="https://github.com/user-attachments/assets/34172bc9-2f94-43cc-a8f0-f5eee7ea990a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update documentation priority weight [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #47529:
URL: https://github.com/apache/airflow/pull/47529#issuecomment-2708035025
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org