You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/07/23 04:47:20 UTC
[I] Provider migration for decoupling remote logging from core [airflow]
jason810496 opened a new issue, #70265:
URL: https://github.com/apache/airflow/issues/70265
- related: #67056
## Why
#67056 decoupled remote logging from the hardcoded if/elif chain in
`airflow-core/src/airflow/config_templates/airflow_local_settings.py`. Core and the Task SDK now
share one resolver (`airflow_shared.logging.factory.resolve_remote_task_log`) that dispatches on the
URL scheme of `[logging] remote_base_log_folder` via `ProvidersManager` and instantiates the
provider class through a no-arg `from_config()` classmethod.
The legacy per-scheme chain in `airflow_local_settings.py` is transitional: it stays as a fallback
until every remote logging backend has been migrated to the new mechanism. This issue tracks that
migration, one sub-issue per backend.
## The pattern
Each migration is a small, self-contained PR with three parts (see the merged amazon examples:
s3 — #69817, cloudwatch — #69816):
1. **Add `from_config()`** — a no-arg classmethod on the provider's `RemoteLogIO` class that mirrors
the corresponding legacy branch in `airflow_local_settings.py`, including the
`[logging] remote_task_handler_kwargs` IO-kwargs merge (where the legacy branch applies it) and
`expanduser` on `base_log_folder`, so behavior is unchanged for existing configs.
2. **Register the scheme** — add a `remote-logging:` section to the provider's `provider.yaml` and
mirror it in `get_provider_info.py`:
```yaml
remote-logging:
- classpath: airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO
scheme: s3
```
3. **Tests** — mirror `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`: `from_config` field mapping,
IO-kwargs filtering, scheme registration, and end-to-end `resolve_remote_task_log` dispatch.
4. **Real system test** — set up the actual backend yourself (a real service or a local
equivalent), run a task with remote logging enabled, and verify logs are uploaded and read back
through the new dispatch path. Include the setup and verification results in the PR description.
If `from_config()` raises on a bad config, the shared factory falls back to the legacy path — so
these migrations are non-breaking by construction.
## Backends
- [x] `s3` (amazon) — #69817
- [x] `cloudwatch` (amazon) — #69816
- [ ] `gs` (google) — #SUB_GS
- [ ] `stackdriver` (google) — #SUB_STACKDRIVER
- [ ] `wasb` (microsoft.azure) — #SUB_WASB
- [ ] `oss` (alibaba) — #SUB_OSS
- [ ] `hdfs` (apache.hdfs) — #SUB_HDFS
- [ ] `elasticsearch` — #SUB_ES
- [ ] `opensearch` — #SUB_OS
## Follow-up once backends are migrated
- [ ] Once all provider migrations are merged **and released**, add a deprecation warning to the
legacy branches in `airflow_local_settings.py`, gated on the provider versions that ship
`from_config` (step 2 of the plan in #67056).
- [ ] Remove the hardcoded remote logging config from core in a future minor or major release,
whichever the community prefers.
## How to contribute
Comment on a sub-issue to claim it — one PR per scheme, following the merged amazon examples.
The `elasticsearch` and `opensearch` sub-issues include a small design decision (those backends are
currently selected by host config rather than URL scheme); design feedback is welcome there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5942634836
All the items tracked here are now closed or merged. Can this be closed, or is there remaining work it should track?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5055046165
@jason810496 I'd like to take this one, thanks!
For the end-to-end verification I'll run a single-node Hadoop container locally with WebHDFS enabled, so no external setup is needed. I'll follow the merged amazon examples (https://github.com/apache/airflow/pull/69817, https://github.com/apache/airflow/pull/69816) and mirror the urlsplit(...).path handling noted in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5401971211
Hey @jason810496.
All 9 backends are now confirmed released, verified at three levels (not just version numbers):
| scheme | package | PR | shipped in |
|---|---|---|---|
| s3, cloudwatch | amazon | #69817, #69816 | 9.33.0 |
| gs | google | #70504 | 22.3.0 |
| stackdriver | google | #66513 | 22.1.0 |
| wasb | microsoft-azure | #70301 | 14.1.0 |
| oss | alibaba | #70682 | 3.4.0 |
| hdfs | apache-hdfs | (merged HDFS migration) | 4.13.0 |
| elasticsearch | elasticsearch | #70525 | 6.9.0 |
| opensearch | opensearch | #70295 | 1.12.0 |
For each: confirmed `from_config` is present in the actual shipped wheel, confirmed the
changelog names that exact PR under that exact version, and confirmed that version is a real upload on PyPI.
Ready to start on the deprecation-warning follow-up whenever you'd like me to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #70265: Provider migration for decoupling remote logging from core
URL: https://github.com/apache/airflow/issues/70265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5581515707
> Hey @jason810496.
All 9 backends are now confirmed released, verified at three levels (not just version numbers):
Sorry for the late reply. Yes, please feel free to raise the PR and tag me to review there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5054657172
Hi @FrankYang0529, @nailo2c,
Would you mind to comment again at each sub-issues? Just in case someone comes across the sub-issue but doesn't take a look at this meta issue.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "Prab-27 (via GitHub)" <gi...@apache.org>.
Prab-27 commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5054880068
I'd like to work on #70269, commented on it , Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5054619440
I would like to try https://github.com/apache/airflow/issues/70272. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5249232747
Hi @baha-bouali ,
I prefer to wait until all those providers be released than add the warning. So that the warning can include the version with provider manager integration supported. Regarding the ES and OS the warning message can contain the next step to configure the compatible upgrade. There isn't much difference even the existing gate check against the `ELASTICSEARCH_HOST`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5246909005
Hey @jason810496!
I'd like to take on the **first follow-up task**, which is adding deprecation warnings to the legacy
branches in `airflow_local_settings.py`.
Now that all 9 backends are merged, my proposed approach will be: a small helper that emits `RemovedInAirflow4Warning` at the top of each of the 7 `elif remote_base_log_folder.startswith(...)` branches (s3, cloudwatch, gs, wasb, stackdriver, oss, hdfs)
2 things I'd like to confirm before making my PR:
1. Should the warning fire unconditionally whenever the branch executes (simpler, message just
tells the user to check their provider version), or should it check the installed provider
version via `importlib.metadata` and only warn if that version already ships `from_config`
(avoids telling someone to "upgrade" when they're already on a new-enough version and
something else is actually wrong)?
2. `elasticsearch`/`opensearch` are gated on `ELASTICSEARCH_HOST`/`OPENSEARCH_HOST` rather than
URL scheme, so they don't fit this pattern the same way.
Thus, I'd propose covering just the 7
scheme-based backends here and leaving ES/OpenSearch for a separate follow-up once their
dispatch design is settled, unless there's a preference for me to look at those too.
Happy to open a draft PR once there's rough agreement on the above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5054633178
Hi, I can work on #70266 and #70267. I have a GCP account that I can use for e2e verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provider migration for decoupling remote logging from core [airflow]
Posted by "Andrushika (via GitHub)" <gi...@apache.org>.
Andrushika commented on issue #70265:
URL: https://github.com/apache/airflow/issues/70265#issuecomment-5054717208
Hi, I would like to take #70268, already commented in that issue. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org