You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/04/24 08:51:12 UTC
[I] Blob querying via Hive fails 2 [hudi]
voonhous opened a new issue, #18577:
URL: https://github.com/apache/hudi/issues/18577
### Bug Description
**What happened:**
**What you expected:**
**Steps to reproduce:**
```scala
// Seed two rows in dt='2024-01-01'. Use OUT_OF_LINE here so subsequent UPDATE/MERGE
// can mutate to a different reference without changing the struct shape between ops
// (avoids mixing INLINE and OUT_OF_LINE branches in a single test).
spark.sql("""
INSERT INTO blob_test VALUES
(1, 'file1', named_struct(
'type', 'OUT_OF_LINE',
'data', cast(null as binary),
'reference', named_struct(
'external_path', 'blobs/seed-1',
'offset', 0L,
'length', 11L,
'managed', false)), '2024-01-01'),
(2, 'file2', named_struct(
'type', 'OUT_OF_LINE',
'data', cast(null as binary),
'reference', named_struct(
'external_path', 'blobs/seed-2',
'offset', 0L,
'length', 9L,
'managed', false)), '2024-01-01')
""")
spark.sql("select id, name, blob_data.type, dt from blob_test").show(10, false)
println("BLOB_SQL_INSERT_SUCCESS")
// UPDATE exercises the BLOB metadata re-attach on the UPDATE write path.
// Per RFC-100 external_path and managed are non-null.
spark.sql("""
UPDATE blob_test
SET blob_data = named_struct(
'type', 'OUT_OF_LINE',
'data', cast(null as binary),
'reference', named_struct(
'external_path', 'blobs/updated-1',
'offset', 10L,
'length', 100L,
'managed', true)),
name = 'file1-updated'
WHERE id = 1
""")
spark.sql("select id, name, blob_data.reference.external_path from blob_test where id = 1").show(10, false)
println("BLOB_SQL_UPDATE_SUCCESS")
// MERGE exercises both MATCHED (UPDATE SET) and NOT MATCHED (INSERT of a new
// row into a new partition dt='2024-01-02'). The USING CTE strips the hudi_type
// metadata from the struct column, so this specifically validates the reattach
// path for MERGE.
spark.sql("""
MERGE INTO blob_test t
USING (
SELECT 2L AS id, 'file2-merged' AS name, named_struct(
'type', 'OUT_OF_LINE',
'data', cast(null as binary),
'reference', named_struct(
'external_path', 'blobs/merged-2',
'offset', 20L,
'length', 200L,
'managed', true)) AS blob_data, '2024-01-01' AS dt
UNION ALL
SELECT 3L AS id, 'file3' AS name, named_struct(
'type', 'OUT_OF_LINE',
'data', cast(null as binary),
'reference', named_struct(
'external_path', 'blobs/inserted-3',
'offset', 300L,
'length', 30L,
'managed', false)) AS blob_data, '2024-01-02' AS dt
) s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.name = s.name, t.blob_data = s.blob_data
WHEN NOT MATCHED THEN INSERT (id, name, blob_data, dt) VALUES (s.id, s.name, s.blob_data, s.dt)
""")
spark.sql("select id, name, blob_data.reference.external_path, dt from blob_test order by id").show(10, false)
println("BLOB_SQL_MERGE_SUCCESS")
// DELETE the NOT MATCHED row we just inserted, returning to 2 rows total.
spark.sql("DELETE FROM blob_test WHERE id = 3")
spark.sql("select id, name, blob_data.type, dt from blob_test order by id").show(10, false)
println("BLOB_SQL_DELETE_SUCCESS")
println("BLOB_SQL_TEST_SUCCESS")
```
Then read with hive using:
```java
// Project a nested struct field through the Hive serde to verify the BLOB
// struct layout round-trips correctly (count() and DESCRIBE do not).
hive.execute(
"SELECT concat('BLOB_TYPE=', blob_data.type) FROM default.blob_test WHERE id = 1")
.expectToSucceed()
.assertStdOutContains("BLOB_TYPE=OUT_OF_LINE");
```
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
Write: Spark3.5.3 + Scala2.12
Read: Hive2.3.10
**Relevant configs:**
### Logs and Stack Trace
```log
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO ql.Driver: Executing command(queryId=root_20260424084452_cbf461b5-9efb-4b04-bb50-8a5680d5a831): SELECT concat('BLOB_TYPE=', blob_data.type) FROM default.blob_test WHERE id = 1
OK
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO ql.Driver: Completed executing command(queryId=root_20260424084452_cbf461b5-9efb-4b04-bb50-8a5680d5a831); Time taken: 0.001 seconds
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO ql.Driver: OK
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO utils.HoodieInputFormatUtils: Reading hoodie metadata from path hdfs://namenode:8020/user/hive/warehouse/blob_test
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO table.HoodieTableConfig: Loading table properties from hdfs://namenode:8020/user/hive/warehouse/blob_test/.hoodie/hoodie.properties
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO table.HoodieTableConfig: Loading table properties from hdfs://namenode:8020/user/hive/warehouse/blob_test/.hoodie/hoodie.properties
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO table.HoodieTableConfig: Loading table properties from hdfs://namenode:8020/user/hive/warehouse/blob_test/.hoodie/metadata/.hoodie/hoodie.properties
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO util.ClusteringUtils: Found 0 files in pending clustering operations
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO metadata.HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
26/04/24 08:44:54 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO hfile.CachingHFileReaderImpl: Initializing global HFileBlockCache with size: 100, TTL: 60 minutes.
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieLogFileReader: Closing Log file reader .files-0000-0_00000000000000000.log.1_0-0-0
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieLogFileReader: Closing Log file reader .files-0000-0_20260424084414466.log.1_3-29-38
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieLogFileReader: Closing Log file reader .files-0000-0_20260424084425785.log.1_3-51-72
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieLogFileReader: Closing Log file reader .files-0000-0_20260424084430227.log.1_3-87-134
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.BaseHoodieLogRecordReader: Ordered instant times seen [00000000000000000, 20260424084414466, 20260424084425785, 20260424084430227, 20260424084435426]
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.BaseHoodieLogRecordReader: Targeted instants that are rolled back are []
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.BaseHoodieLogRecordReader: Total valid instants found are 5. Instants are [00000000000000000, 20260424084414466, 20260424084425785, 20260424084430227, 20260424084435426]
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieLogFileReader: Closing Log file reader .files-0000-0_20260424084435426.log.1_0-109-170
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieMergedLogRecordReader: Number of log files scanned => 0
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO log.HoodieMergedLogRecordReader: Number of entries in Map => 3
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO metadata.BaseTableMetadata: Listed partitions from metadata: #partitions=2
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO metadata.BaseTableMetadata: Listed files in 1 partitions from metadata
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO hudi.BaseHoodieTableFileIndex: [table=blob_test] HoodieFileIndex.listPartitionPathFiles took 8 ms (1 partitions, queryInstant=20260424084435426)
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO util.ClusteringUtils: Found 0 files in pending clustering operations
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO hudi.BaseHoodieTableFileIndex: [table=blob_test] HoodieFileIndex.doRefresh took 921 ms
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO util.TablePathUtils: Getting table path from path: hdfs://namenode:8020/user/hive/warehouse/blob_test/dt=2024-01-01/74e1128a-f3d1-425a-bf26-8c3edb13f14f-0_0-79-109_20260424084430227.parquet
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO util.TablePathUtils: Getting table path from path: hdfs://namenode:8020/user/hive/warehouse/blob_test/dt=2024-01-01/74e1128a-f3d1-425a-bf26-8c3edb13f14f-0_0-79-109_20260424084430227.parquet
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO table.HoodieTableConfig: Loading table properties from hdfs://namenode:8020/user/hive/warehouse/blob_test/.hoodie/hoodie.properties
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: INFO compress.CodecPool: Got brand-new decompressor [.gz]
Failed with exception java.io.IOException:java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
26/04/24 08:44:55 [b31ecc7f-7be4-4719-9303-c8e43b7ca59f main]: ERROR CliDriver: Failed with exception java.io.IOException:java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:523)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:430)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:113)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:78)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:113)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:78)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:66)
at org.apache.hudi.hadoop.HiveRecordContext.lambda$projectRecord$0(HiveRecordContext.java:165)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.common.table.read.HoodieFileGroupReader.next(HoodieFileGroupReader.java:262)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.next(HoodieFileGroupReader.java:342)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.next(HoodieFileGroupReader.java:324)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.hadoop.HoodieFileGroupReaderBasedRecordReader.next(HoodieFileGroupReaderBasedRecordReader.java:192)
at org.apache.hudi.hadoop.HoodieFileGroupReaderBasedRecordReader.next(HoodieFileGroupReaderBasedRecordReader.java:89)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:490)
... 16 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Blob querying via Hive fails 2 [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #18577:
URL: https://github.com/apache/hudi/issues/18577#issuecomment-4311951318
Some observations:
1. Hudi's committed canonical schema, on disk and in **.commit** metadata, **blob_data** is a record named **"blob"** with type: `ENUM("blob_storage_type"), logicalType: "blob"`, and two nullable unions for **data** and **reference.**
2. Hive's HMS-registered schema for the same column, `STRUCT<type STRING, data BINARY, reference STRUCT<...>>`. Hive doesn't know about Hudi logical types, so the hive-sync path flattens BLOB into a plain struct and maps the **enum type** to STRING (you can see this in `hudi-sync/hudi-sync-common/.../SparkSchemaUtils.java, case ENUM: return "\"string\"")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Blob querying via Hive fails 2 [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #18577:
URL: https://github.com/apache/hudi/issues/18577#issuecomment-4341687876
Additional test case to account for: Project a non-leading nested field (reference, pos=2) without also selecting type/data:
```java
hive.execute(
"SELECT concat('BLOB_REF=', blob_data.reference.external_path) "
+ "FROM default.blob_test WHERE id = 1")
.expectToSucceed()
.assertStdOutContains("BLOB_REF=blobs/updated-1");
```
```log
java.io.IOException: java.lang.ClassCastException: class org.apache.hadoop.io.ArrayWritable cannot be cast to class org.apache.hadoop.io.Text (org.apache.hadoop.io.ArrayWritable and org.apache.hadoop.io.Text are in unnamed module of loader 'app')
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:523)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:430)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.ClassCastException: class org.apache.hadoop.io.ArrayWritable cannot be cast to class org.apache.hadoop.io.Text (org.apache.hadoop.io.ArrayWritable and org.apache.hadoop.io.Text are in unnamed module of loader 'app')
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:149)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:78)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:108)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:78)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:108)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:78)
at org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:66)
at org.apache.hudi.hadoop.HiveRecordContext.lambda$projectRecord$0(HiveRecordContext.java:165)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.common.table.read.HoodieFileGroupReader.next(HoodieFileGroupReader.java:262)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.next(HoodieFileGroupReader.java:342)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.next(HoodieFileGroupReader.java:324)
at org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
at org.apache.hudi.hadoop.HoodieFileGroupReaderBasedRecordReader.next(HoodieFileGroupReaderBasedRecordReader.java:192)
at org.apache.hudi.hadoop.HoodieFileGroupReaderBasedRecordReader.next(HoodieFileGroupReaderBasedRecordReader.java:89)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:490)
... 16 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Blob querying via Hive fails 2 [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed issue #18577: Blob querying via Hive fails 2
URL: https://github.com/apache/hudi/issues/18577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org