[PR] Flink 2.3: Preserve nanosecond timestamp filter precision [iceberg]

["openinx (via GitHub)" <gi...@apache.org>]

openinx opened a new pull request, #18498:
URL: https://github.com/apache/iceberg/pull/18498

   Flink SQL filters on TIMESTAMP(9) can discard matching rows because timestamp literals are truncated to microseconds. Preserve higher-precision literals as timestamp strings so filter binding and source split serialization retain all digits.
   
   The SQL example to reproduce this problem: 
   
   ```sql
   SET 'execution.runtime-mode' = 'batch';
   SET 'table.dml-sync' = 'true';
   
   CREATE TABLE trades (
       trade_id INT,
       trade_time TIMESTAMP(9)
   ) WITH (
       'format-version' = '3',
       'write.format.default' = 'parquet'
   );
   
   INSERT INTO trades
   VALUES (
       1,
       CAST('2024-01-01 09:30:00.123456789' AS TIMESTAMP(9))
   );
   
   -- Verify that the trade exists.
   SELECT trade_id, trade_time FROM trades;
   
   -- Look up that exact trade timestamp.
   SELECT trade_id
   FROM trades
   WHERE trade_time =
       CAST('2024-01-01 09:30:00.123456789' AS TIMESTAMP(9));
   
   -- Before the fix:  the query will return no rows. 
   -- After the fix:  the query will return tradeI_id = 1. 
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Flink 2.3: Preserve nanosecond timestamp filter precision [iceberg]

["openinx (via GitHub)" <gi...@apache.org>]

openinx commented on PR #18498:
URL: https://github.com/apache/iceberg/pull/18498#issuecomment-6161434385

   The script to reproduce 
   
   ```bash
   #!/usr/bin/env bash
   set -euo pipefail
   : "${FLINK_HOME:?Set FLINK_HOME to a clean Flink 2.3.0 installation}"
   : "${JAVA_HOME:?Set JAVA_HOME to JDK 17}"
   
   ./gradlew -DsparkVersions= -DkafkaVersions= -DflinkVersions=2.3 \
     --max-workers=2 :iceberg-flink:iceberg-flink-runtime-2.3:shadowJar
   
   REPRO_DIR=$(mktemp -d /tmp/iceberg-nanos.XXXXXX)
   mkdir "$REPRO_DIR/lib" "$REPRO_DIR/log"
   ln -s "$FLINK_HOME"/lib/*.jar "$REPRO_DIR/lib/"
   cp flink/v2.3/flink-runtime/build/libs/iceberg-flink-runtime-2.3-1.13.0-SNAPSHOT.jar \
     "$REPRO_DIR/lib/"
   
   MAVEN_REPO_URL=${MAVEN_REPO_URL:-https://repo.maven.apache.org/maven2}
   for artifact in hadoop-client-api hadoop-client-runtime; do
     curl -fsSL "$MAVEN_REPO_URL/org/apache/hadoop/$artifact/3.4.3/$artifact-3.4.3.jar" \
       -o "$REPRO_DIR/lib/$artifact.jar"
   done
   curl -fsSL "$MAVEN_REPO_URL/commons-logging/commons-logging/1.3.5/commons-logging-1.3.5.jar" \
     -o "$REPRO_DIR/lib/commons-logging.jar"
   
   cat > "$REPRO_DIR/reproduce.sql" <<SQL
   SET 'execution.runtime-mode' = 'batch';
   SET 'table.dml-sync' = 'true';
   SET 'sql-client.execution.result-mode' = 'tableau';
   
   CREATE CATALOG demo WITH (
     'type' = 'iceberg',
     'catalog-type' = 'hadoop',
     'warehouse' = 'file://$REPRO_DIR/warehouse'
   );
   USE CATALOG demo;
   CREATE DATABASE demo_db;
   USE demo_db;
   
   CREATE TABLE trades (trade_id INT, trade_time TIMESTAMP(9)) WITH (
     'format-version' = '3',
     'write.format.default' = 'parquet'
   );
   INSERT INTO trades VALUES
     (1, CAST('2024-01-01 09:30:00.123456789' AS TIMESTAMP(9)));
   
   SELECT trade_id, trade_time FROM trades;
   SELECT trade_id FROM trades
   WHERE trade_time = CAST('2024-01-01 09:30:00.123456789' AS TIMESTAMP(9));
   SQL
   
   FLINK_LIB_DIR="$REPRO_DIR/lib" FLINK_LOG_DIR="$REPRO_DIR/log" \
   JVM_ARGS='-Xmx2g -XX:ActiveProcessorCount=2' \
   "$FLINK_HOME/bin/sql-client.sh" embedded \
     -Dexecution.target=local -Dparallelism.default=1 \
     -hist "$REPRO_DIR/history" -f "$REPRO_DIR/reproduce.sql"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org