You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/06/09 05:59:10 UTC
[PR] Hive: Fix conversion failure for nested VARIANT and reduce type-string allocations [iceberg]
wombatu-kun opened a new pull request, #16735:
URL: https://github.com/apache/iceberg/pull/16735
## Problem
`HiveSchemaUtil.convert(Schema)` builds the Hive column list on every Hive table and view commit (via `HiveOperationsBase.storageDescriptor`). For nested types it recursed through `convert(Type)`, which is `TypeInfoUtils.getTypeInfoFromTypeString(convertToTypeString(type))`: it built the type substring, parsed it back into a Hive `TypeInfo` tree, then stringified that tree straight back to the same substring. That round trip produces no value, and because the recursion went through the parser at every nesting level the wasted work and the throwaway `TypeInfo` allocations compounded with schema depth.
It was also a latent crash. An Iceberg `VARIANT` maps to the placeholder string `"unknown"`, which is not a Hive type, so `TypeInfoUtils.getTypeInfoFromTypeString("unknown")` throws `RuntimeException: Internal error parsing position 0 of 'unknown'`. A `VARIANT` nested inside a `struct`/`list`/`map` therefore failed the commit outright. The top-level case never hit this because `convert(Schema)` calls `convertToTypeString` directly.
## Change
Rewrite the private `convertToTypeString` to recurse on itself via a `StringBuilder`-based `appendTypeString(StringBuilder, Type)`, building the Hive type string directly with no `TypeInfoUtils` parsing and no `String.format`. The public methods and their outputs are unchanged; only the internal string builder is restructured. Output strings are identical for every type the previous code handled, which is pinned byte for byte by the existing `testComplexSchemaConvertToHiveSchema`. Nested `VARIANT` now emits `"unknown"` directly, consistent with the already-tested top-level mapping.
## Tests
- Added `testNestedVariantTypeConvertToHiveSchema`: it throws on the previous code (reproducing the failure) and is green after the change.
- Existing `TestHiveSchemaUtil` coverage (nested struct/list/map strings, every primitive type, reverse conversion) still passes, confirming output is unchanged for all other types.
## Performance
Added `HiveSchemaConversionBenchmark` (JMH). The conversion runs once per commit; the benchmark isolates `HiveSchemaUtil.convert(Schema)`. JDK 17, average time, `-prof gc` (speedup is before/after):
| shape | time before | time after | time speedup | alloc before | alloc after | alloc reduction |
|---|---|---|---|---|---|---|
| flat (30 primitives) | 2.74 us/op | 2.12 us/op | 1.29x | 6,080 B/op | 5,872 B/op | 1.04x |
| nested (struct / list / map columns) | 45.66 us/op | 4.53 us/op | 10.1x | 118,632 B/op | 7,400 B/op | 16.0x |
| deeply nested (10 struct levels) | 40.87 us/op | 0.92 us/op | 44.3x | 122,576 B/op | 2,184 B/op | 56.1x |
Flat schemas are essentially unchanged (primitive cases never used the parsing path); the gain is concentrated on nested schemas, where the throwaway `TypeInfo` trees dominated, and grows with nesting depth (the old code re-parsed each subtree at every level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix schema conversion failure for nested VARIANT [iceberg]
Posted by "spacexm406-web (via GitHub)" <gi...@apache.org>.
spacexm406-web commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-6487010937
Expand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix schema conversion failure for nested VARIANT [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-6486812794
@RussellSpitzer on #15220 you suggested writing a placeholder for variant columns, which #15964 did, but only top-level fields reach it: a variant nested in a struct, list or map still goes through Hive's type parser, which rejects `unknown`, so every commit to such a `HiveCatalog` table fails, `CREATE` included. Could you take a look? The fix bypasses the parser only for children that contain a variant, so no other stored column type changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix schema conversion failure for nested VARIANT [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-5906725751
@huaxingao you approved #15964, which fixed #15220 by mapping `VARIANT` to `unknown` in `HiveSchemaUtil` - could you take a look at this follow-up? Only top-level fields reach that case directly: struct, list and map children still round-trip through Hive's parser, which rejects `unknown`, so a `HiveCatalog` table with a nested variant still fails to commit, `CREATE` included. The fix skips the parser only for children that contain a variant, so every other stored type is unchanged. The red CVE Scan jobs are the repo-wide jackson-databind finding that #18318 fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix conversion failure for nested VARIANT and reduce type-string allocations [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-4931145616
not stale
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix conversion failure for nested VARIANT and reduce type-string allocations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-4930862068
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix conversion failure for nested VARIANT and reduce type-string allocations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-5247537730
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix schema conversion failure for nested VARIANT [iceberg]
Posted by "spacexm406-web (via GitHub)" <gi...@apache.org>.
spacexm406-web commented on PR #16735:
URL: https://github.com/apache/iceberg/pull/16735#issuecomment-6486866538
Initial assessment: Apache Iceberg GitHub comment
Your link points to a specific comment on a pull request in the official Apache Iceberg repository: PR #16735 and its discussion.
The important distinction: a pull request proposes a change to a software project, while an individual comment may contain a technical explanation, a review, a requested correction, or an automated status update.
I attempted to retrieve the linked discussion, but I couldn’t verify the contents of that exact comment. I don’t want to attribute technical claims or conclusions to its author without evidence.
What Apache Iceberg is
Apache Iceberg is an open-source table format for large analytical datasets. It helps data platforms manage huge tables stored in systems such as object storage while supporting features such as:
* Reliable data updates: Track changes to datasets.
* Time travel: Query earlier snapshots of a table.
* Schema evolution: Change table structures without rebuilding everything.
* Query-engine integration: Work with engines such as Spark and Flink.
Official project: Apache Iceberg on GitHub
What I could verify
The repository’s public documentation confirms that Apache Iceberg uses pull requests for code contributions and expects reviewers to assess changes before they are merged.
However, I couldn’t establish what PR #16735 changes or what the specific comment says. I would not draw conclusions about its technical significance, security implications, or acceptance status yet.
Let’s examine the actual comment
Please copy and paste the text of the GitHub comment here, or upload a screenshot of it.
I’ll break it down into:
1. Plain-English meaning: What the author is actually saying.
2. Technical analysis: What the code or proposed change does.
3. Importance: Whether it represents a bug, improvement, security concern, or routine discussion.
4. Verification: What the available evidence confirms and what remains uncertain.
5. Practical value: Whether the underlying idea could be useful for your projects, including OSARETIN PRIME.
That will let me give you a specific, evidence-based assessment rather than speculate about a comment I couldn’t retrieve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org