You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Hueske (Jira)" <ji...@apache.org> on 2026/09/02 16:00:00 UTC
[jira] [Created] (FLINK-40541) LATERAL SNAPSHOT join rejects a valid build-side watermark when it is declared on a column hidden by the column-expansion strategy
Fabian Hueske created FLINK-40541:
-------------------------------------
Summary: LATERAL SNAPSHOT join rejects a valid build-side watermark when it is declared on a column hidden by the column-expansion strategy
Key: FLINK-40541
URL: https://issues.apache.org/jira/browse/FLINK-40541
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Fabian Hueske
Assignee: Fabian Hueske
*Description*
The {{LATERAL SNAPSHOT}} join requires the build-side input to declare exactly one watermark (row-time attribute), which drives the operator's LOAD phase. The planner detects this watermark by counting row-time indicator fields in the row type of the projected {{TABLE}} argument.
When the watermark is declared on a column that the configured column-expansion strategy hides from {{SELECT *}} — e.g. a {{VIRTUAL}} metadata column combined with {{TableConfigOptions.TABLE_COLUMN_EXPANSION_STRATEGY = EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS}} — the {{TABLE t}} argument expands to a projection that drops the row-time attribute. The row-time count is then {{0}}, and the query is wrongly rejected with:
{code}
org.apache.flink.table.api.ValidationException: LATERAL SNAPSHOT requires a watermark on the build-side input.
{code}
even though the table does declare a watermark.
*How to reproduce*
{code:sql}
SET 'table.column-expansion-strategy' = 'EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS';
CREATE TABLE build_side (
bk STRING,
bv INT,
rt TIMESTAMP_LTZ(3) METADATA VIRTUAL,
WATERMARK FOR rt AS rt
) WITH (...);
SELECT * FROM probe
JOIN LATERAL SNAPSHOT(
input => TABLE build_side,
load_completed_condition => 'user_time',
load_completed_time => CAST(TIMESTAMP '2026-07-01 00:00:00' AS TIMESTAMP_LTZ(3))
) AS s ON probe.pk = s.bk;
{code}
Explicitly re-projecting the hidden column (e.g. {{SELECT *, rt FROM build_side}} in a preceding CTE/view) works around the problem, confirming that the watermark itself is present and only the visibility of the row-time column is at fault.
*Proposed fix*
Detect the build-side row-time attribute from the watermark below any projection that hides it, rather than from a visible time-attribute column, and re-append the row-time attribute from below the hiding projection so it reaches the operator. Run the watermark validation on the rewritten build side instead of the raw {{TABLE}} argument. The existing top-level {{Calc}} wrapper projects the re-appended column away, so the user-visible output is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)