You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/08/05 07:32:17 UTC
[PR] Fix version_added for the concurrent pod creation options [airflow]
1fanwang opened a new pull request, #71157:
URL: https://github.com/apache/airflow/pull/71157
The `cncf.kubernetes` options `[kubernetes_executor] async_pod_creation` and
`pod_creation_max_concurrency` are tagged `version_added: 10.20.0`, but they ship in
10.21.0. I added both in https://github.com/apache/airflow/pull/68480 and got the version
wrong.
Two consequences:
* The 10.21.0 configuration reference renders "New in version 10.20.0" for both options,
pointing users at a release where they do not exist.
* `version_added` is also what keeps a newer option off an older version's config page:
`conf_constants.py` drops any option whose `version_added` is greater than the package
version being built. At `10.20.0` the filter does not drop these two, so a 10.20.0 docs
build lists options the 10.20.0 code has never heard of.
Found while verifying `cncf-kubernetes 10.21.0rc1` for
https://github.com/apache/airflow/issues/70953. Three other options in the same section
carry `version_added: 10.20.0` legitimately; only these two are wrong.
The value was right when I wrote it and went stale in review. 10.19.0 was the latest
release when the PR was opened on 2026-06-12, so 10.20.0 was the next version. 10.20.0 was
then cut on 2026-07-23 while the PR was still open, and the PR merged six days later, which
put it in 10.21.0. `version_added` is hand-written and nothing revalidates it: the schema
accepts any string, no pre-commit or breeze check compares it against the version being
released, and the `versions:` list only gains its entry at release-prep time. Any PR that
adds a config option and stays open across a release boundary hits this.
related: https://github.com/apache/airflow/issues/70953
### Testing done
The two options do not exist anywhere in the released 10.20.0 provider, and do exist in
10.21.0rc1. Both wheels installed from PyPI / the RC SVN dist into separate virtualenvs.
<details><summary>Raw logs</summary>
```console
$ .venv/bin/python -c "import importlib.metadata as m; print(m.version('apache-airflow-providers-cncf-kubernetes'))"
10.21.0rc1
$ .venv-prev/bin/python -c "import importlib.metadata as m; print(m.version('apache-airflow-providers-cncf-kubernetes'))"
10.20.0
# every option in the k8s provider.yaml claiming version_added: 10.20.0
$ git show providers-cncf-kubernetes/10.21.0rc1:providers/cncf/kubernetes/provider.yaml > /tmp/rc1_provider.yaml
$ python3 -c "
import yaml
d = yaml.safe_load(open('/tmp/rc1_provider.yaml'))
for sec, body in d['config'].items():
for name, opt in body['options'].items():
if opt.get('version_added') == '10.20.0':
print(f'{sec}.{name}')
"
kubernetes_executor.running_pod_log_lines
kubernetes_executor.async_pod_creation
kubernetes_executor.pod_creation_max_concurrency
kubernetes_executor.pod_launch_failure_retries
kubernetes_executor.pod_launch_failure_excluded_container_reasons
# are those five actually present in the released 10.20.0 provider?
$ for opt in async_pod_creation pod_creation_max_concurrency running_pod_log_lines \
pod_launch_failure_retries pod_launch_failure_excluded_container_reasons; do
n=$(grep -rl "$opt" .venv-prev/lib/python3*/site-packages/airflow/providers/cncf/kubernetes/ | wc -l)
echo "10.20.0 $opt -> $n file(s)"
done
10.20.0 async_pod_creation -> 0 file(s)
10.20.0 pod_creation_max_concurrency -> 0 file(s)
10.20.0 running_pod_log_lines -> 4 file(s)
10.20.0 pod_launch_failure_retries -> 4 file(s)
10.20.0 pod_launch_failure_excluded_container_reasons -> 4 file(s)
# same two options in 10.21.0rc1
$ for opt in async_pod_creation pod_creation_max_concurrency; do
n=$(grep -rl "$opt" .venv/lib/python3*/site-packages/airflow/providers/cncf/kubernetes/ | wc -l)
echo "10.21.0rc1 $opt -> $n file(s)"
done
10.21.0rc1 async_pod_creation -> 7 file(s)
10.21.0rc1 pod_creation_max_concurrency -> 8 file(s)
```
</details>
<details><summary>How the value went stale</summary>
```console
# latest release when the PR was opened on 2026-06-12
$ git show 8e63d9bbc77^:providers/cncf/kubernetes/provider.yaml | grep -A3 '^versions:'
versions:
- 10.19.0
- 10.18.0
- 10.17.1
# 10.20.0 was cut while the PR was open, and the PR merged after it
$ git log -1 --format=%cI 8e63d9bbc77 # the commit adding "- 10.20.0" to versions:
2026-07-23T22:48:43+03:00
$ gh pr view 68480 --repo apache/airflow --json createdAt,mergedAt
created: 2026-06-12T19:09:23Z
merged: 2026-07-29T09:58:47Z
$ git merge-base --is-ancestor 8e63d9bbc77 703a0570c84 && echo "merged after 10.20.0 was cut"
merged after 10.20.0 was cut
# nothing validates the field
$ grep -rn version_added dev/breeze/src/ scripts/ | grep -viE 'jinja|template|\.rst'
(no output)
$ python3 -c "import json; print(json.load(open('airflow-core/src/airflow/provider.yaml.schema.json'))['definitions']['option']['properties']['version_added'])"
{'type': ['string', 'null']}
```
</details>
`get_provider_info.py` carries the same field and is generated from `provider.yaml` by
breeze at package-prep time (`_prepare_get_provider_info_py_file` in
`dev/breeze/src/airflow_breeze/utils/packages.py`), so both are updated together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix version_added for the concurrent pod creation options [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #71157:
URL: https://github.com/apache/airflow/pull/71157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org