You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2025/11/14 11:26:18 UTC
[PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
ayushtkn opened a new pull request, #6185:
URL: https://github.com/apache/hive/pull/6185
### What changes were proposed in this pull request?
For invalid entries get_variant() doesn't return null
### Why are the changes needed?
To correct the behaviour of to_variant()
### Does this PR introduce _any_ user-facing change?
Yes, get_variant() throws instead of returning null
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3542732845
> ohhk, you mean to say the current behaviour is correct, we need not to changed anything?
it's probably correct for the empty path case, but not for the rest of described scenarios. Could we check the same in Spark or ask Spark folks to run these scenarios for us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3541927704
not sure about this. https://issues.apache.org/jira/browse/HIVE-29277 returns NULL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3541977769
in our case: `try_variant_get` should return `null`, in case someone wants `null` he should use `try_variant_get`
```
name = "try_variant_get",
value = "_FUNC_(variant, path[, type]) - Extracts a sub-variant from variant according to path, and casts it to type. Returns null on error.",
extended = """
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3532664149
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6185) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6185&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6185&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6185&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6185&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6185&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6185)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6185:
URL: https://github.com/apache/hive/pull/6185#discussion_r2534132888
##########
iceberg/iceberg-handler/src/test/queries/negative/get_variant_index_non_array.q:
##########
@@ -0,0 +1 @@
+SELECT variant_get(parse_json('{"a": 1}'), '$[0]');
Review Comment:
wouldn't it better to do this in unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3542254716
> I am confused. Isn't this what we fix here: variant_get was returning `null` instead of an error
variant_get(parse_json('{"a": 1}'), '$[0]') should return NULL, not error. I've added examples, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3542113362
I am confused. Isn't this what we fix here: variant_get was returning `null` instead of an error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6185:
URL: https://github.com/apache/hive/pull/6185#discussion_r2534132888
##########
iceberg/iceberg-handler/src/test/queries/negative/get_variant_index_non_array.q:
##########
@@ -0,0 +1 @@
+SELECT variant_get(parse_json('{"a": 1}'), '$[0]');
Review Comment:
wouldn't it better to do this in itest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3542006322
> in our case: `try_variant_get` should return `null`, in case someone wants `null` he should use `try_variant_get`
>
> ```
> name = "try_variant_get",
> value = "_FUNC_(variant, path[, type]) - Extracts a sub-variant from variant according to path, and casts it to type. Returns null on error.",
> extended = """
> ```
✅ variant_get
Throws an error if the extraction fails
Errors happen when:
- JSONPath is invalid (syntax error)
- The value cannot be converted to the requested type
- The variant structure is malformed
If the path is valid but does not match anything, Iceberg returns NULL (same as normal JSONPath behavior)
✔️ Safe for normal use
✖️ Not safe if the data may be malformed
✅ try_variant_get
Never throws an error
If anything goes wrong, it returns NULL
It swallows:
- Invalid JSONPath syntax
- Type conversion failures
- Unexpected data types
- Corrupted or partial JSON/VARIANT values
✔️ Safe for messy/unknown data
✔️ Useful for schema evolution
✔️ Good in SELECT queries where one bad row would otherwise fail the job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3539155303
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6185) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6185&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6185&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6185&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6185&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6185&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6185)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #6185:
URL: https://github.com/apache/hive/pull/6185#issuecomment-3542354308
ohhk, you mean to say the current behaviour is correct, we need not to changed anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn closed pull request #6185: HIVE-29317: Iceberg: get_variant() wrongly return null for invalid entries
URL: https://github.com/apache/hive/pull/6185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org