[I] Add OpensearchRemoteLogIO.from_config and register opensearch scheme [airflow]

["jason810496 (via GitHub)" <gi...@apache.org>]

jason810496 opened a new issue, #70272:
URL: https://github.com/apache/airflow/issues/70272

   Part of #70265 (related: #67056).
   
   ## Why
   
   #67056 decoupled remote logging from the hardcoded branches in `airflow_local_settings.py`: core and
   the Task SDK now resolve the handler via `ProvidersManager` dispatch on the
   `[logging] remote_base_log_folder` URL scheme, instantiating the provider class through a no-arg
   `from_config()` classmethod. This issue migrates the OpenSearch backend.
   
   ## Design note (feedback welcome)
   
   Unlike the object-storage backends, OpenSearch is currently selected by `[opensearch] host` being
   set — `remote_base_log_folder` typically has no scheme for these deployments, so pure scheme
   dispatch cannot reach it. The proposal (same as the sibling Elasticsearch issue #70271):
   
   - Register an `opensearch` scheme and document setting
     `[logging] remote_base_log_folder = opensearch://` as the forward-looking configuration.
   - Keep the legacy host-based selection working via the transitional fallback in
     `airflow_local_settings.py` until the chain is removed (tracked by the meta issue), so existing
     configs are unaffected.
   
   ## What
   
   - [ ] Add `OpensearchRemoteLogIO.from_config()` in
         `providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py`, mirroring the
         legacy branch:
         https://github.com/apache/airflow/blob/104ad12e190db7197fa303d98a3ab68879eafd33/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L349-L379
         — reading the `[opensearch]` options the branch reads (`host`, `port`, `username`,
         `password`, `target_index`, `write_stdout`, `write_to_os`, `json_format`, `host_field`,
         `offset_field`, and `log_id_template` with its
         `{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}` fallback) plus
         `[logging] base_log_folder` (with `expanduser`) and `[logging] delete_local_logs`, so
         behavior is unchanged for existing configs. Note the legacy branch does *not* merge
         `remote_task_handler_kwargs` IO-kwargs for this backend — mirror that too.
   - [ ] Register the `opensearch` scheme under a `remote-logging:` section in
         `providers/opensearch/provider.yaml` and mirror it in
         `providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py`.
   - [ ] Document the `remote_base_log_folder = opensearch://` form in the provider's logging docs.
   - [ ] Add tests mirroring `TestS3RemoteLogIOFromConfig` in
         `providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`.
   - [ ] Verify end to end with a real system test: set up the actual backend yourself (a real
         OpenSearch cluster or a local one, e.g. via Docker), run a task with remote logging enabled,
         and confirm logs are written and read back through the new dispatch path. Include the setup
         and verification results in the PR description.
   
   ## Reference
   
   Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If `from_config` raises on a bad
   config, the shared factory falls back to the legacy path, so this is not a breaking change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add OpensearchRemoteLogIO.from_config and register opensearch scheme [airflow]

["FrankYang0529 (via GitHub)" <gi...@apache.org>]

FrankYang0529 commented on issue #70272:
URL: https://github.com/apache/airflow/issues/70272#issuecomment-5054662455

   I would like to handle this one. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add OpensearchRemoteLogIO.from_config and register opensearch scheme [airflow]

["jason810496 (via GitHub)" <gi...@apache.org>]

jason810496 closed issue #70272: Add OpensearchRemoteLogIO.from_config and register opensearch scheme
URL: https://github.com/apache/airflow/issues/70272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org