You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "Tartarus0zm (via GitHub)" <gi...@apache.org> on 2026/01/09 09:28:47 UTC
[I] Support merge native operators [auron]
Tartarus0zm opened a new issue, #1865:
URL: https://github.com/apache/auron/issues/1865
Merge Native operators that can be chained together into a single chain. This reduces the overhead of row and column conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm closed issue #1865: Support merge native operators
URL: https://github.com/apache/auron/issues/1865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4677015585
@weiqingy Really looking forward to seeing how this turns out.
and we can clearly document the usage and applicable scenarios for Flink on Auron in the user manual.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4677170207
Thanks @Tartarus0zm ! The implementation is up in #2327 (stacked on the small prerequisite fix #2326).
Agreed on documenting the usage and applicable scenarios in the user manual. In short, a Calc fuses into the native Kafka source — running as a single native Project[Filter?[KafkaScan]] pipeline with one columnar→row conversion at the tail — when its input is the native Auron Kafka source, the source has no event-time watermark, and every expression converts. Otherwise the Calc runs as a stock Flink codegen Calc. Event-time-watermark fusion is tracked separately in #2315.
I'll fold this into the Flink-on-Auron docs work (#1892).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4619412771
@weiqingy assigned to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4646762709
Hi @weiqingy thanks for your work!
You raised a really critical point — the watermark handling. From my immediate intuition, if we don't make changes to the Auron Rust code, we can't guarantee the existing watermark constraints. We can track this separately under a dedicated issue, #2315.
For the no-watermark scenario, we still need to make some adjustments before the Kafka source and Calc can be merged together. The Kafka source's output includes three extra metadata columns (Kafka partition, offset, timestamp), so if we connect it directly to the Calc, we'll need to adjust the Calc's input schema to accommodate these three columns. Additionally, all the corresponding column reference indices within the Calc will need to be shifted accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4666967440
Thanks @Tartarus0zm — both points are spot on, and the schema one was a real gap in the design. I've updated the design doc accordingly.
On the metadata columns: you're right that the merge is not a bare leaf swap. The native `KafkaScan` output prepends three metadata columns `[partition, offset, timestamp]` ahead of the logical columns, and the source reads them back by fixed position (`FlinkArrowReader.create(batch, type, 3)` plus the negative-index reads `getInt(-3)`/`getLong(-2)`/`getLong(-1)`) to drive offset-commit and `collectWithTimestamp`. The Calc, on the other hand, converts its Filter/Project `RexInputRef`s 0-based over the logical-only schema. So after swapping the FFIReader leaf for `KafkaScan`, every Calc column reference is off by the metadata count.
The design now reconciles this in `rebaseLeafOntoSource` with `M = 3`: (1) shift every Calc input column ordinal by `+M`; (2) prepend `M` passthrough identity columns (`partition`, `offset`, `timestamp`) to the merged projection so the fused output stays metadata-prefixed; (3) declare the merged output RowType as that metadata-prefixed type. That keeps the source's existing per-record bookkeeping — and the timestamp column the watermark generator will need — completely unchanged, which is exactly what lets us reconnect the watermark path later.
On watermark: agreed, let's keep event-time fusion out of this PR and track it under #2315 (it needs native-engine watermark support to preserve the existing constraints). The projection-only-under-watermark sub-case folds into #2315 as well. This PR fuses only when `watermarkStrategy == null`, and keeps today's FFIReader path otherwise (zero regression).
Two things I'd still like your call on before I write the SPEC:
- Operator-id unification — plan to keep the source's `auron_operator_id` (the offset-commit keys derive from it) and fold the Calc's metrics under it. Sound right?
- "Only convert when input is native" — should a Calc whose input is non-native stop converting to native entirely (revert to a Flink Calc), rather than convert standalone via FFIReader as it does today? That's the stricter, fewer-conversions reading, and it would change the existing #1853 standalone behavior and its tests, so I've left the FFIReader fallback in place for now pending your preference.
Design doc updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4629187469
Thanks, @Tartarus0zm! Here is the drafted design: https://github.com/weiqingy/auron/blob/AURON-1865-design/docs/PR-1865/AURON-1865-DESIGN.md
Could you please help review it when you get a chance? I listed some open questions in the design doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4667245895
hi @weiqingy thanks for your update!
- Operator-id unification ：I agree with keeping the source's `auron_operator_id`.
- Only convert when input is native ：For now, I'm leaning toward this approach — to avoid a situation where new users, unaware of the implementation details, try Flink on Auron and find the performance doesn't meet their expectations, leading them to abandon Flink on Auron altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4619186410
Hi @Tartarus0zm I can start working on this once PR[#2303](https://github.com/apache/auron/pull/2303) is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support merge native operators [auron]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #1865:
URL: https://github.com/apache/auron/issues/1865#issuecomment-4671788877
Thanks @Tartarus0zm — that settles both, and I've folded both into the design.
Operator-id: agreed, the merged operator keeps the source's `auron_operator_id` and folds the Calc's metrics under it.
Strict gating: agreed, and I think it's the right call for exactly the reason you gave — a non-fusible Calc running as a lone native island pays a row→columnar conversion in and a columnar→row out for no fusion win, so it can look worse than a stock Flink Calc and give a misleading first impression. So the design now converts a Calc to native only when it can fuse into a native upstream chain (native input + fully convertible + no event-time watermark); in every other case it stays a stock Flink Calc. The standalone FFIReader native-Calc path is dropped.
One consequence worth flagging: this changes #1853's standalone behavior, so a few of its tests move with it. Cases whose Calc input is the native Kafka source with no watermark now assert the fused plan (`Project[Filter[KafkaScan]]`, no separate Calc operator) — row sets unchanged; cases feeding a Calc from a non-native input now assert the Flink-Calc fallback. I'll reconcile `StreamExecCalcTest` / `AuronFlinkCalcITCase` / `AuronCalcRewriteITCase` as part of the implementation.
Design doc updated. I'll start implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org