You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 09:25:16 UTC
[I] Type mismatch issue when Spark writes to Hudi and synchronizes metadata [hudi]
hudi-bot opened a new issue, #16327:
URL: https://github.com/apache/hudi/issues/16327
When I write Hudi data through Spark, when synchronizing data, I encounter situations where SmallInt and Byte types cannot be converted to Int.
{code:java}
CREATE TABLE if not exists hudi_ddl_cow_t2
(
qgp string,
sfb byte
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'qgp'
);
insert into $tb values('q1', 1){code}
Throw an exception:
{code:java}
Could not convert field Type from TINYINT to int for field sfb {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7167
- Type: Bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Type mismatch issue when Spark writes to Hudi and synchronizes metadata [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16327:
URL: https://github.com/apache/hudi/issues/16327#issuecomment-3630135833
### Linked PR(s)
- https://github.com/apache/hudi/pull/10229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Type mismatch issue when Spark writes to Hudi and synchronizes metadata [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16327:
URL: https://github.com/apache/hudi/issues/16327#issuecomment-4795581112
This appears to be fixed on current `master`. The catalog/Hudi schema mismatch behind `Could not convert field Type from TINYINT to int` was addressed by [HUDI-6219](https://issues.apache.org/jira/browse/HUDI-6219) (commit `ea547e5681a0`, PR #8725, May 2023): `HoodieCatalogTable.tableSchema` now loads the schema from the Hudi meta directory first, and `CreateHoodieTableCommand` writes that canonical schema into the Spark catalog — so byte/short columns (which Avro/Hudi normalize to INT in `HoodieSparkSchemaConverters.scala`) are stored as INT on both sides, eliminating the mismatch on read. A regression test using `id tinyint` exists at `TestCreateTable.scala` ("Test hoodie table schema consistency for non-Avro data types"). The originally linked PR #10229 was closed without merging, presumably because HUDI-6219 already covered it. Closing as fixed unless you can still reproduce on a recent release — please reopen with the Hudi/Spark versions if so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Type mismatch issue when Spark writes to Hudi and synchronizes metadata [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16327: Type mismatch issue when Spark writes to Hudi and synchronizes metadata
URL: https://github.com/apache/hudi/issues/16327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org