You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "weiqingy (via GitHub)" <gi...@apache.org> on 2026/07/22 04:45:27 UTC

[PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

weiqingy opened a new pull request, #2415:
URL: https://github.com/apache/auron/pull/2415

   # Which issue does this PR close?
   
   Closes #2329
   
   # Rationale for this change
   
   #1865 moved Calc-source fusion to a graph-level pass (`AuronOperatorFusionProcessor`) that counts each source's consumers and fuses only sole-consumer sources. A source feeding more than one Calc (for example `UNION ALL` over one table under Flink's default source reuse) is declined at the sole-consumer gate, and each Calc runs as a standalone native operator over the shared source stream.
   
   That multi-consumer path is correct but was under-tested and under-documented. The existing shared-source test only covered the default config, where object reuse is effectively off so Flink deep-copies each row to heap per consumer. The interesting case is object reuse ON: Flink then hands the same `AuronColumnarRowData` reference to both Calc consumers with no defensive copy in between, which is exactly where a fan-out aliasing or use-after-free bug would surface. Nothing pinned that. And the reason a multi-consumer source declines fusion was only a one-line comment.
   
   # What changes are included in this PR?
   
   A new end-to-end test `testSharedSourceUnionAllFanOutSafeWithObjectReuse` in `AuronKafkaSourceMergeITCase`: the same shared-source `UNION ALL` as the existing test, but with object reuse enabled. It asserts the row set stays correct and both Calcs remain standalone operators (the multi-consumer source is not fused).
   
   An expanded doc-comment at the sole-consumer gate in `AuronOperatorFusionProcessor` explaining why a multi-consumer source is not fused (the native source runtime is single-plan and single-output, so N per-consumer plans cannot share one source) and why declining is safe (each consumer runs a standalone native Calc that copies every column out of the shared columnar view).
   
   No production logic changes.
   
   # Are there any user-facing changes?
   
   No.
   
   # How was this patch tested?
   
   `./build/mvn test -Pspark-3.5 -Pscala-2.12 -Pflink-1.18 -pl auron-flink-extension/auron-flink-planner -am -Dtest=AuronKafkaSourceMergeITCase` with the native library built. All 4 tests pass.
   
   # Was this patch authored or co-authored using generative AI tooling?
   - [x] Yes
   - [ ] No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on PR #2415:
URL: https://github.com/apache/auron/pull/2415#issuecomment-5086014364

   Thanks for the review @slfan1989.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on PR #2415:
URL: https://github.com/apache/auron/pull/2415#issuecomment-5042361325

   Hi @Tartarus0zm, could you please help review this PR when you get a chance? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627596728


##########
auron-flink-extension/auron-flink-planner/src/main/java/org/apache/auron/flink/table/planner/processor/AuronOperatorFusionProcessor.java:
##########
@@ -197,7 +197,12 @@ void tryFuse(
             Map<Integer, Integer> consumerCount,
             Function<StreamExecTableSourceScan, DynamicTableSource> sourceResolver,
             ReadableConfig tableConfig) {
-        // Sole-consumer gate: multi-consumer fusion is out of scope (tracked separately).
+        // Sole-consumer gate: a source feeding more than one consumer is deliberately not fused.
+        // The native source runtime is single-plan / single-output — one source runs exactly one
+        // PhysicalPlanNode and emits one stream, so N distinct per-consumer Calc plans cannot share
+        // a single source. Declining loses nothing structurally: each consumer instead runs as a
+        // standalone native Calc over the source's shared row stream, copying every column out of
+        // the shared columnar RowData view, so the fan-out is safe.

Review Comment:
   Good catch. The unfused Calc can indeed run as Flink codegen. StreamExecCalc.translateToPlanInternal falls back to super when the Calc does not convert to native, so asserting native-copy semantics unconditionally was wrong. Reworded to make the structural point the gate actually protects: no per-consumer plan is staged onto the shared source, so there is no last-write-wins between consumers, whether each consumer ends up native or codegen. Fixed in f525cc00.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627656861


##########
auron-flink-extension/auron-flink-planner/src/test/java/org/apache/auron/flink/table/kafka/AuronKafkaSourceMergeITCase.java:
##########
@@ -140,4 +140,42 @@ public void testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse() {
         rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
         assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
     }
+
+    /**
+     * The same shared-source {@code UNION ALL} as {@link
+     * #testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse}, but with object reuse enabled. Under
+     * object reuse Flink hands the <em>same</em> {@code AuronColumnarRowData} reference to both
+     * standalone Calc consumers with no defensive copy in between (the sibling test runs with reuse
+     * off, where Flink deep-copies the row to heap per consumer). The row set is the empirical
+     * guarantee here: the projections and filters convert, so each consumer runs as a standalone
+     * native Auron Calc that eagerly copies every column out of the shared columnar view into its
+     * own Arrow batch before returning, and neither consumer retains a reference to the reused row.
+     * If a native Calc instead aliased the shared row id or read the batch after it was recycled,
+     * the row set would collapse or corrupt (all rows folding onto the last row of a batch) or fault
+     * on freed off-heap buffers, so the row-set assertion below is what pins that native path safe.
+     *
+     * <p>The gate assertion still holds under object reuse: a multi-consumer source is never fused,
+     * so both Calcs remain standalone operators and {@code calcOperatorCount} is 2.
+     */
+    @Test
+    public void testSharedSourceUnionAllFanOutSafeWithObjectReuse() {
+        environment.setParallelism(1);
+        environment.getConfig().enableObjectReuse();
+        try {
+            String sql =
+                    "SELECT `age` FROM T5 WHERE `age` > 20 " + "UNION ALL SELECT `age` + 1 FROM T5 WHERE `age` > 10";
+
+            assertThat(calcOperatorCount(sql))
+                    .as(
+                            "a shared (multi-consumer) source must not fuse under object reuse; both Calcs must remain operators")
+                    .isEqualTo(2);
+
+            List<Row> rows = CollectionUtil.iteratorToList(
+                    tableEnvironment.executeSql(sql).collect());
+            rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
+            assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
+        } finally {
+            environment.getConfig().disableObjectReuse();
+        }

Review Comment:
   This test toggles ExecutionConfig object reuse on a StreamExecutionEnvironment that is shared across tests (PER_CLASS lifecycle). The finally block unconditionally disables object reuse, which can silently change the pre-test setting if it was already enabled by setup or another test. Capture the previous setting and restore it to keep the test isolated and order-independent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "Tartarus0zm (via GitHub)" <gi...@apache.org>.
Tartarus0zm merged PR #2415:
URL: https://github.com/apache/auron/pull/2415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627541536


##########
auron-flink-extension/auron-flink-planner/src/main/java/org/apache/auron/flink/table/planner/processor/AuronOperatorFusionProcessor.java:
##########
@@ -197,7 +197,12 @@ void tryFuse(
             Map<Integer, Integer> consumerCount,
             Function<StreamExecTableSourceScan, DynamicTableSource> sourceResolver,
             ReadableConfig tableConfig) {
-        // Sole-consumer gate: multi-consumer fusion is out of scope (tracked separately).
+        // Sole-consumer gate: a source feeding more than one consumer is deliberately not fused.
+        // The native source runtime is single-plan / single-output — one source runs exactly one
+        // PhysicalPlanNode and emits one stream, so N distinct per-consumer Calc plans cannot share
+        // a single source. Declining loses nothing structurally: each consumer instead runs as a
+        // standalone native Calc over the source's shared row stream, copying every column out of
+        // the shared columnar RowData view, so the fan-out is safe.

Review Comment:
   This comment claims that declining fusion for multi-consumer sources is safe because each consumer runs a standalone *native* Calc that copies every column out of the shared columnar RowData view. However, in this method (and per the tryFuse Javadoc), the unfused Calc may also run as Flink codegen when it is not convertible to native (or when the shadowed StreamExecCalc is not present). The comment should avoid asserting native-copy semantics unconditionally and instead phrase the safety argument conditionally for the native path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627607815


##########
auron-flink-extension/auron-flink-planner/src/test/java/org/apache/auron/flink/table/kafka/AuronKafkaSourceMergeITCase.java:
##########
@@ -140,4 +140,41 @@ public void testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse() {
         rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
         assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
     }
+
+    /**
+     * The same shared-source {@code UNION ALL} as {@link
+     * #testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse}, but with object reuse enabled. Under
+     * object reuse Flink hands the <em>same</em> {@code AuronColumnarRowData} reference to both
+     * standalone Calc consumers with no defensive copy in between (the sibling test runs with reuse
+     * off, where Flink deep-copies the row to heap per consumer). The fan-out still produces the
+     * correct row set because each Calc eagerly copies every column out of the shared columnar view
+     * into its own Arrow batch before returning, so neither consumer retains a reference to the
+     * reused row. If the consumers instead aliased the shared row id or read the batch after it was
+     * recycled, the row set would collapse or corrupt (all rows folding onto the last row of a
+     * batch) or fault on freed off-heap buffers.

Review Comment:
   This test comment explains the fan-out safety in terms of each Calc copying columns into its own Arrow batch, but the test does not assert that the UNION ALL Calcs actually run via the standalone *native* Calc path (it only asserts they remain standalone Calcs). To avoid baking in an unconditional native-copy explanation, reword this to be conditional on the native Calc conversion path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627648012


##########
auron-flink-extension/auron-flink-planner/src/test/java/org/apache/auron/flink/table/kafka/AuronKafkaSourceMergeITCase.java:
##########
@@ -140,4 +140,41 @@ public void testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse() {
         rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
         assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
     }
+
+    /**
+     * The same shared-source {@code UNION ALL} as {@link
+     * #testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse}, but with object reuse enabled. Under
+     * object reuse Flink hands the <em>same</em> {@code AuronColumnarRowData} reference to both
+     * standalone Calc consumers with no defensive copy in between (the sibling test runs with reuse
+     * off, where Flink deep-copies the row to heap per consumer). The fan-out still produces the
+     * correct row set because each Calc eagerly copies every column out of the shared columnar view
+     * into its own Arrow batch before returning, so neither consumer retains a reference to the
+     * reused row. If the consumers instead aliased the shared row id or read the batch after it was
+     * recycled, the row set would collapse or corrupt (all rows folding onto the last row of a
+     * batch) or fault on freed off-heap buffers.

Review Comment:
   Good catch. You're right that the assertions pin non-fusion (`count == 2`) and the row set, not that the standalone Calcs run through the native path.
   
   Reworded in c12805bb: the Arrow-batch copy is now framed as the native Auron Calc behavior the projections and filters convert to, and the comment names the row-set assertion as the empirical guarantee for that path instead of stating the copy as an unconditional fact.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AURON #2329] Test and document multi-consumer source-Calc fan-out safety [auron]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #2415:
URL: https://github.com/apache/auron/pull/2415#discussion_r3627691822


##########
auron-flink-extension/auron-flink-planner/src/test/java/org/apache/auron/flink/table/kafka/AuronKafkaSourceMergeITCase.java:
##########
@@ -140,4 +140,42 @@ public void testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse() {
         rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
         assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
     }
+
+    /**
+     * The same shared-source {@code UNION ALL} as {@link
+     * #testSharedSourceUnionAllDoesNotFuseUnderDefaultReuse}, but with object reuse enabled. Under
+     * object reuse Flink hands the <em>same</em> {@code AuronColumnarRowData} reference to both
+     * standalone Calc consumers with no defensive copy in between (the sibling test runs with reuse
+     * off, where Flink deep-copies the row to heap per consumer). The row set is the empirical
+     * guarantee here: the projections and filters convert, so each consumer runs as a standalone
+     * native Auron Calc that eagerly copies every column out of the shared columnar view into its
+     * own Arrow batch before returning, and neither consumer retains a reference to the reused row.
+     * If a native Calc instead aliased the shared row id or read the batch after it was recycled,
+     * the row set would collapse or corrupt (all rows folding onto the last row of a batch) or fault
+     * on freed off-heap buffers, so the row-set assertion below is what pins that native path safe.
+     *
+     * <p>The gate assertion still holds under object reuse: a multi-consumer source is never fused,
+     * so both Calcs remain standalone operators and {@code calcOperatorCount} is 2.
+     */
+    @Test
+    public void testSharedSourceUnionAllFanOutSafeWithObjectReuse() {
+        environment.setParallelism(1);
+        environment.getConfig().enableObjectReuse();
+        try {
+            String sql =
+                    "SELECT `age` FROM T5 WHERE `age` > 20 " + "UNION ALL SELECT `age` + 1 FROM T5 WHERE `age` > 10";
+
+            assertThat(calcOperatorCount(sql))
+                    .as(
+                            "a shared (multi-consumer) source must not fuse under object reuse; both Calcs must remain operators")
+                    .isEqualTo(2);
+
+            List<Row> rows = CollectionUtil.iteratorToList(
+                    tableEnvironment.executeSql(sql).collect());
+            rows.sort(Comparator.comparingInt(o -> (int) o.getField(0)));
+            assertThat(rows).isEqualTo(Arrays.asList(Row.of(21), Row.of(21), Row.of(22), Row.of(22), Row.of(23)));
+        } finally {
+            environment.getConfig().disableObjectReuse();
+        }

Review Comment:
   Good point. The environment is shared across the class (PER_CLASS, created once in `@BeforeAll`), so an unconditional disable would clobber the setting if it were ever enabled globally.
   
   Fixed in fdd685b1: capture `isObjectReuseEnabled()` before enabling and restore it in the finally block, so the test stays order-independent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org