You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Joy-2000 (via GitHub)" <gi...@apache.org> on 2026/10/26 12:42:45 UTC

[I] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Joy-2000 opened a new issue, #20438:
URL: https://github.com/apache/hudi/issues/20438

   ### Describe the problem you faced
   
   [20261026-184822.svg.zip](https://github.com/user-attachments/files/33965524/20261026-184822.svg.zip)
   
   While investigating the CPU cost of a Spark query over a Hudi Copy-on-Write (COW) Parquet table, we found two `UnsafeProjection` stages on the scan path:
   
   | Sampled stack | Inclusive samples | Share of 7,258 samples |
   | --- | ---: | ---: |
   | Hudi `HoodieFileGroupReaderBasedFileFormat.projectIter -> SpecificUnsafeProjection.apply` | 812 | 11.19% |
   | Spark `FileSourceScanExec.doExecute -> SpecificUnsafeProjection.apply` | 291 | 4.01% |
   
   Both have generated `writeFields_*` calls beneath them. The scan also contains `VectorizedParquetRecordReader` and `VectorizedColumnReader`, indicating vectorized Parquet decoding while this plan supplies rows upstream. We have not observed a wrong query result or an exception.
   
   Could maintainers help clarify what each projection is required to do in this COW row-output path? In particular, can a row already projected by Hudi pass through Spark without another full-row materialization when its schema, column order, representation, and lifetime satisfy Spark's requirements? The profile shows two stages, but does not establish that every row is copied twice or that either stage is redundant.
   
   ### To Reproduce
   
   1. Set spark.sql.codegen.maxFields=30
   2. Read all columns of a cow table (like TPCDS catalog_sales table) whose fields greater than 30;
   3. Capture a task CPU flame graph
   
   Query SQL:
   ```
   SELECT
       sum(length(cs_sold_time_sk)),
       sum(length(cs_ship_date_sk)),
       sum(length(cs_bill_customer_sk)),
       sum(length(cs_bill_cdemo_sk)),
   sum(length(cs_bill_hdemo_sk)),
       sum(length(cs_bill_addr_sk)),
       sum(length(cs_ship_customer_sk)),
       sum(length(cs_ship_cdemo_sk)),
       sum(length(cs_ship_hdemo_sk)),
       sum(length(cs_ship_addr_sk)),
       sum(length(cs_call_center_sk)),
       sum(length(cs_catalog_page_sk)),
       sum(length(cs_ship_mode_sk)),
       sum(length(cs_warehouse_sk)),
       sum(length(cs_item_sk)),
       sum(length(cs_promo_sk)),
       sum(length(cs_order_number)),
       sum(length(cs_quantity)),
       sum(length(cs_wholesale_cost)),
       sum(length(cs_list_price)),
       sum(length(cs_sales_price)),
       sum(length(cs_ext_discount_amt)),
       sum(length(cs_ext_sales_price)),
       sum(length(cs_ext_wholesale_cost)),
       sum(length(cs_ext_list_price)),
   sum(length(cs_ext_tax)),
       sum(length(cs_coupon_amt)),
       sum(length(cs_ext_ship_cost)),
       sum(length(cs_net_paid)),
       sum(length(cs_net_paid_inc_tax)),
       sum(length(cs_net_paid_inc_ship)),
       sum(length(cs_net_paid_inc_ship_tax)),
       sum(length(cs_net_profit)),
       sum(length(cs_sold_date_sk))
   FROM tmp.tpcds_sf1000_catalog_sales;
   ```
   
   ### Expected behavior
   
   If Hudi's and Spark's projections serve the same purpose and produce equivalent rows with the required schema, column order, values, and lifetime, we expect one of the two materializations to be avoidable. If both are necessary, we would like to understand their distinct responsibilities and whether there is another safe way to reduce the row-projection cost.
   
   ### Environment Description
   
   * Hudi version: 1.1.1
   * Spark version: 3.4
   * Flink version:
   * Hive version:
   * Hadoop version:
   * Storage (HDFS/S3/GCS..):
   * Running on Docker? (yes/no):
   
   
   ### Additional context
   
   _No response_
   
   ### Stacktrace
   
   ```shell
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #20438:
URL: https://github.com/apache/hudi/issues/20438#issuecomment-6459023679

   @Joy-2000 Thanks for sharing the details! Your screenshots show that projectIter is called even when there are no partition columns. In the [official Hudi 1.1.1 implementation](https://github.com/apache/hudi/blob/release-1.1.1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L384-L417), that branch returns the Parquet reader directly. 
   
   Have you customized this code? Is the build in the screenshots the same one used to capture the flame graph?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #20438:
URL: https://github.com/apache/hudi/issues/20438#issuecomment-6459766396

   closed as an regression from internal changes instead of oss.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #20438:
URL: https://github.com/apache/hudi/issues/20438#issuecomment-6435778627

   Thanks for the profile. In Hudi 1.1.1, the sampled `HoodieFileGroupReaderBasedFileFormat.projectIter` is called by the [base-file read path](https://github.com/apache/hudi/blob/release-1.1.1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L399-L417) when some partition columns are read from the file and others are supplied from partition values.
   
   For comparison, the ordinary non-bootstrap COW snapshot path in Hudi 0.14.1 without schema-on-read generally reads the partition columns either all from the file or all from partition values. It does not have this mixed-partition `projectIter` stage, although Spark may still perform its own row conversion.
   
   The 1.1.1 Hudi projection may be needed to reorder the partition columns. However, if `from == to` and the reader returns a `ColumnarBatchRow`, Hudi still materializes it as an `UnsafeRow`, even though Spark may subsequently perform another conversion.
   
   Could you share:
   
   - The ordered partition fields, key generator, ordering/precombine field, and partition read options.
   - If possible, the `from` and `to` schemas passed to `projectIter`, including field order and types, and the input row class.
   
   These would help determine whether the Hudi projection reorders columns in your case or is an identity projection.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Posted by "Joy-2000 (via GitHub)" <gi...@apache.org>.
Joy-2000 commented on issue #20438:
URL: https://github.com/apache/hudi/issues/20438#issuecomment-6443171174

   @cshuo Thanks for the detailed analysis and the comparison with 0.14.1.
   
   I checked the profiled Hudi 1.1.1 run. In this invocation of `projectIter`, `from == to` is true: the fields, their order, and their types are identical. The input row class is `ColumnarBatchRow`. Thus this Hudi projection does not reorder columns; it materializes the columnar row view as an `UnsafeRow`.
   
   The table configuration is:
   - No partition fields
   - No precombine field (I have not confirmed whether a separate ordering field is configured)
   - `hoodie.table.recordkey.fields=cs_item_sk,cs_order_number`
   - `hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator`
   
   I have not yet confirmed the exact partition-read options. Given the nonpartitioned table configuration, I am also unsure why the mixed-partition branch is reached. Could an effective `partitionSchema` be present here despite the table configuration? If Hudi's work here is only `ColumnarBatchRow` to `UnsafeRow` materialization, could this conversion or Spark's subsequent `UnsafeProjection` be avoided safely?
   
   Debug screenshots of the schemas and row type:
   
   <img width="2560" height="1379" alt="Image" src="https://github.com/user-attachments/assets/8daa6a2c-a0af-4cc4-bfa1-a919f8e6b13c" />
   
   <img width="5120" height="2832" alt="Image" src="https://github.com/user-attachments/assets/4bb77a10-27b3-478d-b6ae-16df89c95401" />
   
   <img width="2560" height="1417" alt="Image" src="https://github.com/user-attachments/assets/c605cd98-4412-438c-84fe-dcfbc1c6f24f" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #20438: [Support] COW scan shows repeated UnsafeProjection in Hudi and Spark row paths
URL: https://github.com/apache/hudi/issues/20438


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org