You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Hueske (Jira)" <ji...@apache.org> on 2026/09/02 16:00:00 UTC

[jira] [Created] (FLINK-40541) LATERAL SNAPSHOT join rejects a valid build-side watermark when it is declared on a column hidden by the column-expansion strategy

Fabian Hueske created FLINK-40541:
-------------------------------------

             Summary: LATERAL SNAPSHOT join rejects a valid build-side watermark when it is declared on a column hidden by the column-expansion strategy
                 Key: FLINK-40541
                 URL: https://issues.apache.org/jira/browse/FLINK-40541
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / Planner
            Reporter: Fabian Hueske
            Assignee: Fabian Hueske


*Description*

The {{LATERAL SNAPSHOT}} join requires the build-side input to declare exactly one watermark (row-time attribute), which drives the operator's LOAD phase. The planner detects this watermark by counting row-time indicator fields in the row type of the projected {{TABLE}} argument.

When the watermark is declared on a column that the configured column-expansion strategy hides from {{SELECT *}} — e.g. a {{VIRTUAL}} metadata column combined with {{TableConfigOptions.TABLE_COLUMN_EXPANSION_STRATEGY = EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS}} — the {{TABLE t}} argument expands to a projection that drops the row-time attribute. The row-time count is then {{0}}, and the query is wrongly rejected with:

{code}
org.apache.flink.table.api.ValidationException: LATERAL SNAPSHOT requires a watermark on the build-side input.
{code}

even though the table does declare a watermark.

*How to reproduce*

{code:sql}
SET 'table.column-expansion-strategy' = 'EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS';

CREATE TABLE build_side (
  bk STRING,
  bv INT,
  rt TIMESTAMP_LTZ(3) METADATA VIRTUAL,
  WATERMARK FOR rt AS rt
) WITH (...);

SELECT * FROM probe
  JOIN LATERAL SNAPSHOT(
    input => TABLE build_side,
    load_completed_condition => 'user_time',
    load_completed_time => CAST(TIMESTAMP '2026-07-01 00:00:00' AS TIMESTAMP_LTZ(3))
  ) AS s ON probe.pk = s.bk;
{code}

Explicitly re-projecting the hidden column (e.g. {{SELECT *, rt FROM build_side}} in a preceding CTE/view) works around the problem, confirming that the watermark itself is present and only the visibility of the row-time column is at fault.

*Proposed fix*

Detect the build-side row-time attribute from the watermark below any projection that hides it, rather than from a visible time-attribute column, and re-append the row-time attribute from below the hiding projection so it reaches the operator. Run the watermark validation on the rewritten build side instead of the raw {{TABLE}} argument. The existing top-level {{Calc}} wrapper projects the re-appended column away, so the user-visible output is unchanged.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)