You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/08/02 11:56:09 UTC

[I] [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query [hudi]

wombatu-kun opened a new issue, #19457:
URL: https://github.com/apache/hudi/issues/19457

### Bug Description

**What happened:**

A Hudi table whose schema evolved `float` -> `double` fails the whole query through the Trino connector as soon as a predicate on the evolved column is pushed down, for as long as any base file still stores that column as parquet `FLOAT`:

```
TrinoException: Malformed Parquet file. Corrupted statistics for column "[c7] optional float c7"
```

The mechanism sits in trino-parquet, not in Hudi. `TupleDomainParquetPredicate` picks its branch on the type of the pushed-down DOMAIN, which is the Trino type the metastore column handle carries, and then casts the parquet statistics values to it:

```java
// lib/trino-parquet/src/main/java/io/trino/parquet/predicate/TupleDomainParquetPredicate.java, Trino 481
Domain domain = getDomain(column, effectivePredicateDomain.getType(), ...);
...
if (type.equals(DOUBLE)) {
Double min = (Double) minimums.get(i);
Double max = (Double) maximums.get(i);
```

After the evolution the metastore says `DOUBLE` while the base file still holds `FLOAT`, so the statistics values are `Float` and the cast throws `ClassCastException`. `corruptionException(...)` wraps it into a `ParquetCorruptionException`, and `HudiPageSourceProvider.createPageSource` maps that to `HUDI_BAD_DATA`.

The integer family is benign by contrast: `TupleDomainParquetPredicate.asLong` accepts any `Byte`/`Short`/`Integer`/`Long`, so an `int -> long` promotion reads fine. It is specifically the float/double family that breaks.

**What you expected:**

The predicate to be evaluated against the promoted type, or the domain to be dropped for base files that predate the evolution, so the query returns rows instead of failing. Reading the same column without a predicate on it already works.

**Steps to reproduce:**
1. Write a Hudi table with a `float` column, so at least one base file stores it as parquet `FLOAT`.
2. Evolve that column to `double` and sync the metastore, leaving the existing base files unrewritten.
3. Query the table through the Trino connector with a predicate on that column.

Reported behaviour, measured during the review of #19456:

| configuration | result |
|---|---|
| `hudi.parquet.use-column-names=true` | fails, naming the evolved column |
| `hudi.parquet.use-column-names=false` | fails, naming the evolved column |
| before #19456, `hudi.parquet.use-column-names=false` | fails, naming an unrelated column that happened to sit at the stale metastore ordinal |

This is not a regression from #19456, which rebuilt pushed-down predicate handles on physical file ordinals. Positional mode failed before that change too, only while naming the wrong column; #19456 strictly improves the diagnostics here without addressing the underlying failure.

### Environment

**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Trino (hudi-trino connector, Trino 481)
**Relevant configs:** `hudi.parquet.use-column-names` (either value)

### Logs and Stack Trace

```
io.trino.spi.TrinoException: Malformed Parquet file. Corrupted statistics for column "[c7] optional float c7"
Caused by: java.lang.ClassCastException: class java.lang.Float cannot be cast to class java.lang.Double
at io.trino.parquet.predicate.TupleDomainParquetPredicate.getDomain(TupleDomainParquetPredicate.java)
```

### Coverage gap

`hudi-trino` carries no schema-evolution test at all today, in either column-resolution mode, which is why this goes unnoticed. A fixture writing a base file under one type while registering the promoted type in the metastore would pin both the float/double failure and the benign int/long promotion.

Follow-up from the review of #19456. Context: #18780.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.

wombatu-kun commented on issue #19457:
URL: https://github.com/apache/hudi/issues/19457#issuecomment-5162950140

Schema-on-write.

`HoodieSchemaCompatibilityChecker.getCompatibility`, reached from `HoodieTable.validateSchema()` as `checkReaderWriterCompatibility(writerSchema, tableSchema, true)`, permits `long <- int`, `float <- int|long`, `double <- int|long|float`, `string <- any numeric` and `bytes <- string`. `double <- float` is the case here. `hoodie.avro.schema.validate` also defaults to `false`, so by default that check does not even run.

The connector cannot tell the two paths apart here in any case: it takes the column type from the Hive metastore and the physical type from the parquet footer, and never looks at the internal schema. `hudi-trino` carries no `InternalSchema` reference at all today, which is what #19381 is about.

So the two are complementary. #19381 is about resolution, which file column a query column maps to and how to decode it. This one is about the predicate, whether a domain may be compared against that column's statistics at all. The guard stays necessary after #19381 lands unless that work also transcodes domains into the file's physical type; its type table already covers the schema-on-read-only promotions for when it does.

PR: https://github.com/apache/hudi/pull/19467

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.

voonhous commented on issue #19457:
URL: https://github.com/apache/hudi/issues/19457#issuecomment-5162641503

   @wombatu-kun Is this route of schema-evolution schema-on-write or schema-on-read?
   
   If this is a schema-on-write problem, then yes, we need to fix this. 
   
   We have a schema-on-read evolution path that we need to handle too, which i have marked as a todo here:
   https://github.com/apache/hudi/issues/19381
   
   Given that there are 2 schema evolution paths now, we definitely need to unify this moving forward so that things area easier to maintain.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.

voonhous closed issue #19457: [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query
URL: https://github.com/apache/hudi/issues/19457


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org