[jira] [Created] (SPARK-60123) Reading ORC tables with collated STRING columns fails

["Yibo Dong (Jira)" <ji...@apache.org>]

Yibo Dong created SPARK-60123:
---------------------------------

             Summary: Reading ORC tables with collated STRING columns fails
                 Key: SPARK-60123
                 URL: https://issues.apache.org/jira/browse/SPARK-60123
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
         Environment: Spark: 5.0.0-SNAPSHOT
Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
            Reporter: Yibo Dong


h2. What happened

Spark can create and write an ORC table with a {{STRING COLLATE UTF8_LCASE}} column, but reading the stored values fails with an exception. The same column type works with Parquet.

h2. How to reproduce

Run the following SQL using {{spark-sql}}:

{code:sql}
CREATE TABLE t_parquet_collated_string_control (
  k STRING COLLATE UTF8_LCASE
) USING PARQUET;

INSERT INTO t_parquet_collated_string_control VALUES ('a'), ('A'), ('b');

SELECT k FROM t_parquet_collated_string_control WHERE k = 'a';

CREATE TABLE t_orc_collated_string (
  k STRING COLLATE UTF8_LCASE
) USING ORC;

INSERT INTO t_orc_collated_string VALUES ('a'), ('A'), ('b');

SELECT k FROM t_orc_collated_string;
{code}

Save the SQL as {{repro.sql}} and run:

{code:bash}
spark-sql --master 'local[2]' 
  --conf spark.ui.enabled=false 
  --conf spark.sql.catalogImplementation=in-memory 
  --conf "spark.sql.warehouse.dir=$(mktemp -d)" 
  -f repro.sql
{code}

h2. Expected result

The Parquet query should return {{a}} and {{A}}, since {{UTF8_LCASE}} provides case-insensitive comparison.

The ORC query should return all three inserted values: {{a}}, {{A}}, and {{b}}.

h2. Actual result

The Parquet query returns {{a}} and {{A}} as expected. The ORC query fails with:

{code:text}
org.apache.spark.SparkException:
[FAILED_READ_FILE.NO_HINT] Encountered error while reading file
file:///.../part-....orc. SQLSTATE: KD001

Caused by: java.lang.IllegalArgumentException:
Can't parse category at 'struct<`k`:string collate UTF^8_LCASE>'
    at org.apache.orc.impl.ParserUtils.parseCategory(ParserUtils.java:70)
    at org.apache.orc.TypeDescription.fromString(TypeDescription.java:230)
    at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:142)
{code}

h2. Additional information

The failure also occurs with an equality filter, so it is not specific to filter pushdown.

The likely cause is that {{OrcUtils.getOrcSchemaString}} uses Spark's {{catalogString}} for a collated {{StringType}}, producing a schema string that ORC's {{TypeDescription}} parser cannot handle.

The existing [CollationSuite test|https://github.com/apache/spark/blob/38422aefaf599ffb4cfa579d98898d95b1f02081/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala#L491-L510] covers table creation and insertion for file-based data sources, including ORC, but does not read back the stored column values.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org