You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/02/17 05:02:00 UTC

[jira] [Updated] (SPARK-55568) Separate schema construction from field statistics collection

     [ https://issues.apache.org/jira/browse/SPARK-55568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-55568:
-----------------------------------
    Labels: pull-request-available  (was: )

> Separate schema construction from field statistics collection
> -------------------------------------------------------------
>
>                 Key: SPARK-55568
>                 URL: https://issues.apache.org/jira/browse/SPARK-55568
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.1.0, 4.1.1
>            Reporter: Qiegang Long
>            Priority: Major
>              Labels: pull-request-available
>
> Variant shredding schema inference is expensive and can take well over 100ms per file. Propose an optimization to separate field stats collection and schema construction to eliminate repeated schema merge and intermediate allocations



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org