You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@liminal.apache.org by "dependabot[bot] (via GitHub)" <gi...@apache.org> on 2023/05/08 20:29:36 UTC
[GitHub] [incubator-liminal] dependabot[bot] opened a new pull request, #106: Bump apache-airflow from 2.1.2 to 2.6.0
dependabot[bot] opened a new pull request, #106:
URL: https://github.com/apache/incubator-liminal/pull/106
Bumps [apache-airflow](https://github.com/apache/airflow) from 2.1.2 to 2.6.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/apache/airflow/releases">apache-airflow's releases</a>.</em></p>
<blockquote>
<h2>Apache Airflow 2.6.0</h2>
<h2>Significant Changes</h2>
<h3>Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (<a href="https://redirect.github.com/apache/airflow/issues/29506">#29506</a>).</h3>
<p>Default setting handles case where impersonation is needed and both users (airflow and the impersonated user)
have the same group set as main group. Previously the default was also other-writeable and the user might choose
to use the other-writeable setting if they wish by configuring <code>file_task_handler_new_folder_permissions</code>
and <code>file_task_handler_new_file_permissions</code> in <code>logging</code> section.</p>
<h3>SLA callbacks no longer add files to the dag processor manager's queue (<a href="https://redirect.github.com/apache/airflow/issues/30076">#30076</a>)</h3>
<p>This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU,
and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state.</p>
<h3>The <code>cleanup()</code> method in BaseTrigger is now defined as asynchronous (following async/await) pattern (<a href="https://redirect.github.com/apache/airflow/issues/30152">#30152</a>).</h3>
<p>This is potentially a breaking change for any custom trigger implementations that override the <code>cleanup()</code>
method and uses synchronous code, however using synchronous operations in cleanup was technically wrong,
because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays
impacting other triggers. The change is unlikely to affect any existing trigger implementations.</p>
<h3>The gauge <code>scheduler.tasks.running</code> no longer exist (<a href="https://redirect.github.com/apache/airflow/issues/30374">#30374</a>)</h3>
<p>The gauge has never been working and its value has always been 0. Having an accurate
value for this metric is complex so it has been decided that removing this gauge makes
more sense than fixing it with no certainty of the correctness of its value.</p>
<h3>Consolidate handling of tasks stuck in queued under new <code>task_queued_timeout</code> config (<a href="https://redirect.github.com/apache/airflow/issues/30375">#30375</a>)</h3>
<p>Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations
responsible for timing out stuck queued tasks have been deprecated and merged into
<code>[scheduler] task_queued_timeout</code>. The configurations that have been deprecated are
<code>[kubernetes] worker_pods_pending_timeout</code>, <code>[celery] stalled_task_timeout</code>, and
<code>[celery] task_adoption_timeout</code>. If any of these configurations are set, the longest timeout will be
respected. For example, if <code>[celery] stalled_task_timeout</code> is 1200, and <code>[scheduler] task_queued_timeout</code>
is 600, Airflow will set <code>[scheduler] task_queued_timeout</code> to 1200.</p>
<h2>Improvement Changes</h2>
<h3>Display only the running configuration in configurations view (<a href="https://redirect.github.com/apache/airflow/issues/28892">#28892</a>)</h3>
<p>The configurations view now only displays the running configuration. Previously, the default configuration
was displayed at the top but it was not obvious whether this default configuration was overridden or not.
Subsequently, the non-documented endpoint <code>/configuration?raw=true</code> is deprecated and will be removed in
Airflow 3.0. The HTTP response now returns an additional <code>Deprecation</code> header. The <code>/config</code> endpoint on
the REST API is the standard way to fetch Airflow configuration programmatically.</p>
<h3>Explicit skipped states list for ExternalTaskSensor (<a href="https://redirect.github.com/apache/airflow/issues/29933">#29933</a>)</h3>
<p>ExternalTaskSensor now has an explicit <code>skipped_states</code> list</p>
<h2>Miscellaneous Changes</h2>
<h3>Handle OverflowError on exponential backoff in next_run_calculation (<a href="https://redirect.github.com/apache/airflow/issues/28172">#28172</a>)</h3>
<p>Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via <code>core.max_task_retry_delay</code>
parameter.</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/apache/airflow/blob/main/RELEASE_NOTES.rst">apache-airflow's changelog</a>.</em></p>
<blockquote>
<h2>Airflow 2.6.0 (2023-04-30)</h2>
<p>Significant Changes
^^^^^^^^^^^^^^^^^^^</p>
<p>Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (<a href="https://redirect.github.com/apache/airflow/issues/29506">#29506</a>).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Default setting handles case where impersonation is needed and both users (airflow and the impersonated user)
have the same group set as main group. Previously the default was also other-writeable and the user might choose
to use the other-writeable setting if they wish by configuring <code>file_task_handler_new_folder_permissions</code>
and <code>file_task_handler_new_file_permissions</code> in <code>logging</code> section.</p>
<p>SLA callbacks no longer add files to the dag processor manager's queue (<a href="https://redirect.github.com/apache/airflow/issues/30076">#30076</a>)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU,
and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state.</p>
<p>The <code>cleanup()</code> method in BaseTrigger is now defined as asynchronous (following async/await) pattern (<a href="https://redirect.github.com/apache/airflow/issues/30152">#30152</a>).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This is potentially a breaking change for any custom trigger implementations that override the <code>cleanup()</code>
method and uses synchronous code, however using synchronous operations in cleanup was technically wrong,
because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays
impacting other triggers. The change is unlikely to affect any existing trigger implementations.</p>
<p>The gauge <code>scheduler.tasks.running</code> no longer exist (<a href="https://redirect.github.com/apache/airflow/issues/30374">#30374</a>)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The gauge has never been working and its value has always been 0. Having an accurate
value for this metric is complex so it has been decided that removing this gauge makes
more sense than fixing it with no certainty of the correctness of its value.</p>
<p>Consolidate handling of tasks stuck in queued under new <code>task_queued_timeout</code> config (<a href="https://redirect.github.com/apache/airflow/issues/30375">#30375</a>)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations
responsible for timing out stuck queued tasks have been deprecated and merged into
<code>[scheduler] task_queued_timeout</code>. The configurations that have been deprecated are
<code>[kubernetes] worker_pods_pending_timeout</code>, <code>[celery] stalled_task_timeout</code>, and
<code>[celery] task_adoption_timeout</code>. If any of these configurations are set, the longest timeout will be
respected. For example, if <code>[celery] stalled_task_timeout</code> is 1200, and <code>[scheduler] task_queued_timeout</code>
is 600, Airflow will set <code>[scheduler] task_queued_timeout</code> to 1200.</p>
<p>Improvement Changes
^^^^^^^^^^^^^^^^^^^</p>
<p>Display only the running configuration in configurations view (<a href="https://redirect.github.com/apache/airflow/issues/28892">#28892</a>)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The configurations view now only displays the running configuration. Previously, the default configuration
was displayed at the top but it was not obvious whether this default configuration was overridden or not.
Subsequently, the non-documented endpoint <code>/configuration?raw=true</code> is deprecated and will be removed in
Airflow 3.0. The HTTP response now returns an additional <code>Deprecation</code> header. The <code>/config</code> endpoint on</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/apache/airflow/commit/ab54c63940a99646df974d4bcf2e37415e277e69"><code>ab54c63</code></a> fixup! Update Release Note</li>
<li><a href="https://github.com/apache/airflow/commit/af9ca33ac2e642045319310a1b7421a5c377eca7"><code>af9ca33</code></a> Fix static_folder for cli app (<a href="https://redirect.github.com/apache/airflow/issues/30952">#30952</a>)</li>
<li><a href="https://github.com/apache/airflow/commit/07272b62f17754cdaa4ca6b1fa67e9a85a894214"><code>07272b6</code></a> Update Release Note</li>
<li><a href="https://github.com/apache/airflow/commit/e2e8f127d4224d22882c0cfc96faa032420b3443"><code>e2e8f12</code></a> Initialize plugins for cli appbuilder (<a href="https://redirect.github.com/apache/airflow/issues/30934">#30934</a>)</li>
<li><a href="https://github.com/apache/airflow/commit/82548bad5c2fd57bfe3ec5e0982f3e059e581279"><code>82548ba</code></a> fixup! Update Release Note</li>
<li><a href="https://github.com/apache/airflow/commit/568ca2b4c70af430d81ee725ae3bb3681f99b954"><code>568ca2b</code></a> Fix dag file processor hertbeat to run only if necessary (<a href="https://redirect.github.com/apache/airflow/issues/30899">#30899</a>)</li>
<li><a href="https://github.com/apache/airflow/commit/37026566b9e553b17907ec91634ab5edf4c5ca94"><code>3702656</code></a> Update Release Note</li>
<li><a href="https://github.com/apache/airflow/commit/695e437829d8a21ef15ce24deec57d7f50f3312c"><code>695e437</code></a> Fix KubernetesExecutor sending state to scheduler (<a href="https://redirect.github.com/apache/airflow/issues/30872">#30872</a>)</li>
<li><a href="https://github.com/apache/airflow/commit/cbadaf6eaa2b91afbc2614e0e928f6e6a6c47153"><code>cbadaf6</code></a> Rename most pod_id usage to pod_name in KubernetesExecutor (<a href="https://redirect.github.com/apache/airflow/issues/29147">#29147</a>)</li>
<li><a href="https://github.com/apache/airflow/commit/cf5c3c8d02f0089adfcfa3dad9e2580d290251e6"><code>cf5c3c8</code></a> Fix failing trigger test after Flask 2.2.4 upgrade (<a href="https://redirect.github.com/apache/airflow/issues/30875">#30875</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/apache/airflow/compare/2.1.2...2.6.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-liminal/network/alerts).
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] dependabot[bot] commented on pull request #106: Bump apache-airflow from 2.1.2 to 2.6.0
Posted by "dependabot[bot] (via GitHub)" <gi...@apache.org>.
dependabot[bot] commented on PR #106:
URL: https://github.com/apache/incubator-liminal/pull/106#issuecomment-1632953635
Superseded by #108.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] dependabot[bot] closed pull request #106: Bump apache-airflow from 2.1.2 to 2.6.0
Posted by "dependabot[bot] (via GitHub)" <gi...@apache.org>.
dependabot[bot] closed pull request #106: Bump apache-airflow from 2.1.2 to 2.6.0
URL: https://github.com/apache/incubator-liminal/pull/106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org