[jira] [Created] (FLINK-35341) Retraction stop working with Clock dependent function in Filter

["Lim Qing Wei (Jira)" <ji...@apache.org>]

Lim Qing Wei created FLINK-35341:
------------------------------------

             Summary: Retraction stop working with Clock dependent function in Filter
                 Key: FLINK-35341
                 URL: https://issues.apache.org/jira/browse/FLINK-35341
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / Runtime
    Affects Versions: 1.17.1
            Reporter: Lim Qing Wei


 

Say we have a Flink SQL view where
 # we use clock dependent function like `UNIX_TIMESTAMP()` in query filter, eg. WHERE clause, eg. table.timestamp < UNIX_TIMESTAMP()
 # source record is retracted at a time where the filter is evaluated as false

we expect a retraction is produced from the view, but in practice nothing happen.

 

We are using kafka as a source, here's a small snippet that shows the problem.

 
{code:java}
CREATE TEMPORARY VIEW my_view AS
    SELECT key,
            someData,
            expiry
    FROM upstream 
    WHERE expiry > UNIX_TIMESTAMP();


select * from my_view where key = 5574332;{code}
 

 

The actual query is a bit more complicated but this simplified one should illustrate the issue. Below is the event happen in chronological order:

 
 # Run this query as a stream
 # Create a record in upstream where key = 5574332, and expiry to be in 3 minutes into the future.
 # Observe insertion of the record, as expected
 # Wait for 3 minutes
 # Now the record should expired, but given there's no update, there's no change to the stream output just yet
 # Delete the upstream record (using tombstone in kafka)
 # Observe no change in the stream output, but we are expecting retraction(aka deletion)

 

Is this a known issue? I've search Jira but could find any, I observed this in 1.15 until 1.17, havent tested with 1.18 and above though



--
This message was sent by Atlassian Jira
(v8.20.10#820010)