You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2026/07/21 11:22:16 UTC

[PR] Add OpenLineage parent info to EMR Spark steps [airflow]

mobuchowski opened a new pull request, #70182:
URL: https://github.com/apache/airflow/pull/70182

   `EmrAddStepsOperator` can now pass OpenLineage parent job metadata to classic EMR Spark steps. The new `openlineage_inject_parent_job_info` option defaults to `openlineage.spark_inject_parent_job_info`, matching the other Spark-based Amazon operators.
   
   The injection reuses OpenLineage's Spark property helper and translates its output into tokenized `--conf key=value` arguments for `command-runner.jar` steps using `spark-submit` or `run-example`. It leaves non-Spark steps, custom Jar steps, and steps with existing parent settings unchanged, and does not mutate the operator's original step configuration.
   
   This lets OpenLineage events emitted by Spark link back to the originating Airflow task run when parent information injection is enabled.
   
   Validation:
   
   - `uv run --project providers/amazon pytest providers/amazon/tests/unit/amazon/aws/operators/test_emr_add_steps.py -xvs` — 15 passed
   - `prek run --from-ref main --stage pre-commit` — passed
   - `breeze ci selective-check --commit-ref 2ea76a5ef7` — selected the expected Amazon/provider CI coverage
   - The manual provider mypy hook and selected parallel provider suite could not start because the local Buildah CI-image build rejected its configured cache reference; both stopped before running checks.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Codex (GPT-5)
   
   Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   Drafted-by: Codex (GPT-5) (no human review before posting)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add OpenLineage parent info to EMR Spark steps [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #70182:
URL: https://github.com/apache/airflow/pull/70182


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org