[I] [Bug] Spark cannot read __paimon_bucket from chain table merge results [paimon]

["taoran92 (via GitHub)" <gi...@apache.org>]

taoran92 opened a new issue, #10651:
URL: https://github.com/apache/paimon/issues/10651

   ### Search before asking
   
   - [x] I searched existing issues and found no report of this chain-specific failure. #10478 describes a related failure involving the record iterator type on regular primary-key tables; this issue involves the split type used for chain reads.
   
   ### Paimon version
   
   Paimon 2.2-SNAPSHOT.
   
   The affected code is present on master at commit 702810b32b1b0c7189e31191e9234fca3d107c32.
   
   ### Compute Engine
   
   Spark 3.5.9, Scala 2.12, Java 11.
   
   ### Minimal reproduce step
   
   When a chain table query merges snapshot and delta data, selecting ordinary columns succeeds. Selecting __paimon_bucket, or using it only in a filter, fails.
   
   Configure a Paimon catalog named spark_test with the Paimon Spark extensions enabled:
   
   ```properties
   spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkCatalog
   spark.sql.catalog.spark_test.warehouse=file:/tmp/paimon-chain-bucket-repro
   spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
   ```
   
   Run the following SQL in a fresh test database:
   
   ```sql
   CREATE DATABASE spark_test.chain_bucket_repro;
   USE spark_test.chain_bucket_repro;
   
   CREATE TABLE T (
       id INT,
       payload STRING,
       seq BIGINT,
       dt STRING
   )
   USING paimon
   PARTITIONED BY (dt)
   TBLPROPERTIES (
       'primary-key' = 'dt,id',
       'bucket-key' = 'id',
       'bucket' = '1',
       'merge-engine' = 'deduplicate',
       'sequence.field' = 'seq',
       'chain-table.enabled' = 'true',
       'partition.timestamp-pattern' = '$dt',
       'partition.timestamp-formatter' = 'yyyyMMdd'
   );
   
   CALL spark_test.sys.create_branch('chain_bucket_repro.T', 'snapshot');
   CALL spark_test.sys.create_branch('chain_bucket_repro.T', 'delta');
   
   ALTER TABLE T SET TBLPROPERTIES (
       'scan.fallback-snapshot-branch' = 'snapshot',
       'scan.fallback-delta-branch' = 'delta'
   );
   
   ALTER TABLE `T$branch_snapshot` SET TBLPROPERTIES (
       'scan.fallback-snapshot-branch' = 'snapshot',
       'scan.fallback-delta-branch' = 'delta'
   );
   
   ALTER TABLE `T$branch_delta` SET TBLPROPERTIES (
       'scan.fallback-snapshot-branch' = 'snapshot',
       'scan.fallback-delta-branch' = 'delta'
   );
   
   -- Two rows in the snapshot.
   INSERT INTO `T$branch_snapshot`
   VALUES
       (1, 'old', 1, '20250810'),
       (2, 'keep', 1, '20250810');
   
   -- Update one row in the next day's delta.
   INSERT INTO `T$branch_delta`
   VALUES
       (1, 'new', 2, '20250811');
   
   -- Succeeds: returns (1, 'new') and (2, 'keep').
   SELECT id, payload
   FROM T
   WHERE dt = '20250811';
   
   -- Fails: bucket metadata in the projection.
   SELECT id, __paimon_bucket
   FROM T
   WHERE dt = '20250811';
   
   -- Also fails: bucket metadata only in the filter.
   SELECT id, payload
   FROM T
   WHERE dt = '20250811'
     AND __paimon_bucket = 0;
   ```
   
   Run the two failing queries separately if the SQL client stops after an error.
   
   ### What doesn't meet your expectations?
   
   Both queries involving bucket metadata should succeed. The table has one bucket, so all records belong to bucket 0.
   
   Expected result for the metadata projection, ignoring row order:
   
   ```text
   id  __paimon_bucket
   1   0
   2   0
   ```
   
   Expected result for the bucket filter:
   
   ```text
   id  payload
   1   new
   2   keep
   ```
   
   Instead, both queries fail with:
   
   ```text
   Caused by: java.lang.RuntimeException: There need be DataSplit when path and index metadata columns are required
       at org.apache.paimon.spark.PaimonRecordReaderIterator.<init>(PaimonRecordReaderIterator.scala:45)
       at org.apache.paimon.spark.PaimonPartitionReader.readSplit(PaimonPartitionReader.scala:121)
       at org.apache.paimon.spark.PaimonPartitionReader.<init>(PaimonPartitionReader.scala:54)
       at org.apache.paimon.spark.PaimonPartitionReaderFactory.createReader(PaimonPartitionReaderFactory.scala:47)
       at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:125)
   ```
   
   ### Anything else?
   
   The chain merge path produces ChainSplit, wrapped as FallbackSplitImpl(ChainSplit) by the fallback scan. PaimonPartitionReader passes that split to PaimonRecordReaderIterator.
   
   When partition or bucket metadata is requested, the iterator requires the split to be a DataSplit. Neither FallbackSplitImpl nor ChainSplit extends DataSplit, so initialization fails before returning any rows.
   
   ```text
   Snapshot DataSplits + Delta DataSplits
                     |
             Group files by bucket
                     |
                 ChainSplit
                     |
       FallbackSplitImpl(ChainSplit)
                     |
       Spark metadata reader requires DataSplit
                     |
                RuntimeException
   ```
   
   Chain splits are constructed per bucket, so the merged rows still have a well-defined bucket ID. The failure occurs in the Spark metadata adapter.
   
   The exception message mentions “path and index”, but the failing check actually applies to partition and bucket metadata. A possible fix is to inspect the underlying split and support partition/bucket metadata from ChainSplit, while preserving the original wrapper for reading and authorization.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Spark cannot read __paimon_bucket from chain table merge results [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #10651: [Bug] Spark cannot read __paimon_bucket from chain table merge results
URL: https://github.com/apache/paimon/issues/10651


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org