You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/09/19 21:51:44 UTC
[GitHub] [hive] ayushtkn opened a new pull request, #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
ayushtkn opened a new pull request, #3607:
URL: https://github.com/apache/hive/pull/3607
### What changes were proposed in this pull request?
Temporary Hack to allow for non mixed ORC tables to support Vectorization
### How was this patch tested?
Added Explain vectorization at relevant places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1262873809
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1258936683
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976701086
##########
iceberg/iceberg-handler/src/test/queries/positive/vectorized_iceberg_read_orc.q:
##########
@@ -20,14 +22,17 @@ create external table tbl_ice_orc_all_types (
t_date DATE,
t_decimal DECIMAL(4,2)
) stored by iceberg stored as orc;
-
-insert into tbl_ice_orc_all_types values (1.1, 1.2, false, 4, 567890123456789, '6', "col7", cast('2012-10-03 19:58:08' as timestamp), date('1234-09-09'), cast('10.01' as decimal(4,2)));
+--casting throws ClassCastException, Disable untill fixed.
Review Comment:
if it's a known issue, please create JIRA and link it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976701086
##########
iceberg/iceberg-handler/src/test/queries/positive/vectorized_iceberg_read_orc.q:
##########
@@ -20,14 +22,17 @@ create external table tbl_ice_orc_all_types (
t_date DATE,
t_decimal DECIMAL(4,2)
) stored by iceberg stored as orc;
-
-insert into tbl_ice_orc_all_types values (1.1, 1.2, false, 4, 567890123456789, '6', "col7", cast('2012-10-03 19:58:08' as timestamp), date('1234-09-09'), cast('10.01' as decimal(4,2)));
+--casting throws ClassCastException, Disable untill fixed.
Review Comment:
if it's a known issue, please create JIRA and mention it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1258523719
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r984264418
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -177,9 +178,19 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
@Override
public VectorizedSupport.Support[] getSupportedFeatures() {
- // disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
- // beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ throw new UnsupportedOperationException("This overload of getSupportedFeatures should never be called");
+ }
+
+ @Override
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties) {
+ // disabling VectorizedSupport.Support.DECIMAL_64 for Parquet as it doesn't support it
+ boolean isORCOnly = Boolean.parseBoolean(properties.getProperty(HiveIcebergMetaHook.ORC_FILES_ONLY));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r983438232
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -125,6 +125,7 @@ public class HiveIcebergMetaHook implements HiveMetaHook {
private static final List<org.apache.commons.lang3.tuple.Pair<Integer, byte[]>> EMPTY_FILTER =
Lists.newArrayList(org.apache.commons.lang3.tuple.Pair.of(1, new byte[0]));
static final String MIGRATED_TO_ICEBERG = "MIGRATED_TO_ICEBERG";
+ static final String ORC_FILES_ONLY = "orc.files.only";
Review Comment:
This is probably ends being an Iceberg-Hive integration logic only, and thus perhaps we should prepend this with "iceberg" so it's `iceberg.orc.files.only` or something similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1257841989
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1251997555
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r981798738
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java:
##########
@@ -18,11 +18,15 @@
package org.apache.hadoop.hive.ql.exec.vector;
+import org.apache.hadoop.hive.conf.HiveConf;
+
+import java.util.Properties;
+
/**
* Marker interface to indicate a given input format supports
* vectorization input.
*/
public interface VectorizedInputFormatInterface {
- VectorizedSupport.Support[] getSupportedFeatures();
+ VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties);
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r983452649
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -177,9 +178,19 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
@Override
public VectorizedSupport.Support[] getSupportedFeatures() {
- // disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
- // beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ throw new UnsupportedOperationException("This overload of getSupportedFeatures should never be called");
+ }
+
+ @Override
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties) {
+ // disabling VectorizedSupport.Support.DECIMAL_64 for Parquet as it doesn't support it
+ boolean isORCOnly = Boolean.parseBoolean(properties.getProperty(HiveIcebergMetaHook.ORC_FILES_ONLY));
Review Comment:
We also need another check here: verify if write.format.default is ORC. If this is changed from within Hive or anything using our HMS client then of course ORC_FILES_ONLY is maintained, but if changed from just Iceberg API it will not reflect the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r980905190
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java:
##########
@@ -18,11 +18,15 @@
package org.apache.hadoop.hive.ql.exec.vector;
+import org.apache.hadoop.hive.conf.HiveConf;
+
+import java.util.Properties;
+
/**
* Marker interface to indicate a given input format supports
* vectorization input.
*/
public interface VectorizedInputFormatInterface {
- VectorizedSupport.Support[] getSupportedFeatures();
+ VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties);
Review Comment:
To reduce the scope of the required refactoring. We could add this as a new method while keeping the old one too.
```
VectorizedSupport.Support[] getSupportedFeatures();
+ default VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties) {
+ return getSupportedFeatures();
+ }
```
And then in HiveIcebergInputFormat we could have something like:
```
+ public VectorizedSupport.Support[] getSupportedFeatures() {
+ throw new UnsupportedOperationException("This overload of getSupportedFeatures should never be called");
+ }
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties) {
+ //do the logic you have implemented
+ }
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976709923
##########
iceberg/iceberg-handler/src/test/results/positive/alter_table_to_iceberg.q.out:
##########
@@ -103,6 +103,7 @@ Table Parameters:
metadata_location hdfs://### HDFS PATH ###
numFiles 1
numRows 5
+ orc.files.only false
Review Comment:
I'm not sure if I get the intention of orc.files.only property, but tbl_orc clearly looks like an orc table, isn't it the case? I can see that it has no partitions (so I cannot imagine how can we store other data in other formats here)
so if orc.files.only evaluates to false, we'll miss vectorization opportunity, right?
##########
iceberg/iceberg-handler/src/test/results/positive/alter_table_to_iceberg.q.out:
##########
@@ -103,6 +103,7 @@ Table Parameters:
metadata_location hdfs://### HDFS PATH ###
numFiles 1
numRows 5
+ orc.files.only false
Review Comment:
I'm not sure if I get the intention of "orc.files.only" property, but tbl_orc clearly looks like an orc table, isn't it the case? I can see that it has no partitions (so I cannot imagine how can we store other data in other formats here)
so if orc.files.only evaluates to false, we'll miss vectorization opportunity, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1254301891
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r977158394
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -1264,10 +1266,25 @@ private Support[] getVectorizedInputFormatSupports(
try {
InputFormat inputFormat = FetchOperator.getInputFormatFromCache(inputFileFormatClass, hiveConf);
if (inputFormat instanceof VectorizedInputFormatInterface) {
+ // Special handling for Iceberg tables with ORC format.
+ if (inputFormat.getClass().getName().contains("HiveIcebergInputFormat")) {
+ boolean onlyOrcFiles = true;
+ Hive hiveDb = Hive.get(hiveConf, false);
+ for (String tableName : currentOperator.getSchema().getTableNames()) {
+ boolean isORCOnly =
+ "TRUE".equalsIgnoreCase(hiveDb.getTable(tableName).getParameters().get("orc.files.only"));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976724376
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -1264,10 +1266,25 @@ private Support[] getVectorizedInputFormatSupports(
try {
InputFormat inputFormat = FetchOperator.getInputFormatFromCache(inputFileFormatClass, hiveConf);
if (inputFormat instanceof VectorizedInputFormatInterface) {
+ // Special handling for Iceberg tables with ORC format.
+ if (inputFormat.getClass().getName().contains("HiveIcebergInputFormat")) {
+ boolean onlyOrcFiles = true;
+ Hive hiveDb = Hive.get(hiveConf, false);
+ for (String tableName : currentOperator.getSchema().getTableNames()) {
+ boolean isORCOnly =
+ "TRUE".equalsIgnoreCase(hiveDb.getTable(tableName).getParameters().get("orc.files.only"));
Review Comment:
nit: might this look better? (not sure, depends on taste)
```
Boolean.valueOf(hiveDb.getTable(tableName).getParameters().get("orc.files.only"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976719135
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -1264,10 +1266,25 @@ private Support[] getVectorizedInputFormatSupports(
try {
InputFormat inputFormat = FetchOperator.getInputFormatFromCache(inputFileFormatClass, hiveConf);
if (inputFormat instanceof VectorizedInputFormatInterface) {
+ // Special handling for Iceberg tables with ORC format.
+ if (inputFormat.getClass().getName().contains("HiveIcebergInputFormat")) {
+ boolean onlyOrcFiles = true;
+ Hive hiveDb = Hive.get(hiveConf, false);
+ for (String tableName : currentOperator.getSchema().getTableNames()) {
+ boolean isORCOnly =
+ "TRUE".equalsIgnoreCase(hiveDb.getTable(tableName).getParameters().get("orc.files.only"));
+ if (!isORCOnly) {
+ onlyOrcFiles = false;
Review Comment:
I guess we should break the loop here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1251789560
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1262638519
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1263174289
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r979686170
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -176,10 +179,27 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
}
@Override
- public VectorizedSupport.Support[] getSupportedFeatures() {
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Set<String> tableNames) {
// disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
// beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ boolean onlyOrcFiles = true;
+ try {
+ Hive hiveDb = Hive.get(hiveConf, false);
Review Comment:
I agree that doing HMS calls from InputFormat is not a nice pattern, but I'm not really fond of hardcoding Iceberg related stuff to hive-exec either.
Can't we refactor Vectorizer instead so that supplies the Table object or its properties to getSupportedFeatures()? Perhaps we could get this info from `verifyAndSetVectorPartDesc` through `addVectorizedInputFileFormatSupport` and finally through `getVectorizedInputFormatSupports` ?
It's a bigger refactor but the responsibility of deciding decimal64 or other vector features would remain solely within InputFormats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn merged pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn merged PR #3607:
URL: https://github.com/apache/hive/pull/3607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r983436133
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -177,9 +178,19 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
@Override
public VectorizedSupport.Support[] getSupportedFeatures() {
- // disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
- // beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ throw new UnsupportedOperationException("This overload of getSupportedFeatures should never be called");
+ }
+
+ @Override
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Properties properties) {
+ // disabling VectorizedSupport.Support.DECIMAL_64 for Parquet as it doesn't support it
+ boolean isORCOnly = Boolean.parseBoolean(properties.getProperty(HiveIcebergMetaHook.ORC_FILES_ONLY));
+ if (!isORCOnly) {
+ HiveConf.setVar(hiveConf, HiveConf.ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED, "");
+ return new VectorizedSupport.Support[] {};
+ }
+ HiveConf.setVar(hiveConf, HiveConf.ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED, "decimal_64");
Review Comment:
While in line 189 we do need to set this to empty string to prevent LLAP from picking up decimal64 as supported feature in case HiveIcebergInputformat.getSupportedFeatures is [] we don't want to set this true explicitly here. A user may want to have this disabled, and while the Inputformat could support it we don't want to force the usage of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976692639
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java:
##########
@@ -826,6 +834,18 @@ private Type.PrimitiveType getPrimitiveTypeOrThrow(FieldSchema field) throws Met
return (Type.PrimitiveType) newType;
}
+ private void setOrcOnlyFilesParam(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
+ if (org.apache.iceberg.FileFormat.ORC.name().equalsIgnoreCase(hmsTable.getSd().getInputFormat()) ||
Review Comment:
nit: what about refactoring this huge condition to a method? (it can be even reused later)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r977158651
##########
iceberg/iceberg-handler/src/test/queries/positive/vectorized_iceberg_read_orc.q:
##########
@@ -20,14 +22,17 @@ create external table tbl_ice_orc_all_types (
t_date DATE,
t_decimal DECIMAL(4,2)
) stored by iceberg stored as orc;
-
-insert into tbl_ice_orc_all_types values (1.1, 1.2, false, 4, 567890123456789, '6', "col7", cast('2012-10-03 19:58:08' as timestamp), date('1234-09-09'), cast('10.01' as decimal(4,2)));
+--casting throws ClassCastException, Disable untill fixed.
Review Comment:
No longer an issue now, Sorted in this PR. :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r976709923
##########
iceberg/iceberg-handler/src/test/results/positive/alter_table_to_iceberg.q.out:
##########
@@ -103,6 +103,7 @@ Table Parameters:
metadata_location hdfs://### HDFS PATH ###
numFiles 1
numRows 5
+ orc.files.only false
Review Comment:
I'm not sure if I get the intention of "orc.files.only" property, but tbl_orc clearly looks like an orc table, isn't it the case? I can see that it has no partitions (so I cannot imagine how can we store data in other formats here)
so if orc.files.only evaluates to false, we'll miss vectorization opportunity, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r974755315
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -176,10 +179,27 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
}
@Override
- public VectorizedSupport.Support[] getSupportedFeatures() {
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Set<String> tableNames) {
// disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
// beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ boolean onlyOrcFiles = true;
+ try {
+ Hive hiveDb = Hive.get(hiveConf, false);
Review Comment:
Is it possible to make this decision in Vectorizer::getVectorizedInputFormatSupports? Basically trying to avoid the mixup of HMS related information in InputFormats, as they will be used in AppMasters and tasks. I agree that getSupportedFeatures will be called mainly during compilation, but it will be good to keep HMS interactions outside of inputformats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r975489411
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java:
##########
@@ -176,10 +179,27 @@ public boolean shouldSkipCombine(Path path, Configuration conf) {
}
@Override
- public VectorizedSupport.Support[] getSupportedFeatures() {
+ public VectorizedSupport.Support[] getSupportedFeatures(HiveConf hiveConf, Set<String> tableNames) {
// disabling VectorizedSupport.Support.DECIMAL_64 as Parquet doesn't support it, and we have no way of telling
// beforehand what kind of file format we're going to hit later
- return new VectorizedSupport.Support[]{ };
+ boolean onlyOrcFiles = true;
+ try {
+ Hive hiveDb = Hive.get(hiveConf, false);
Review Comment:
Thanx Rajesh, I have changed it. Just a bit of hardcoding required for the iceberg related stuff, since hive-exec doesn't have iceberg dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r977158477
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -1264,10 +1266,25 @@ private Support[] getVectorizedInputFormatSupports(
try {
InputFormat inputFormat = FetchOperator.getInputFormatFromCache(inputFileFormatClass, hiveConf);
if (inputFormat instanceof VectorizedInputFormatInterface) {
+ // Special handling for Iceberg tables with ORC format.
+ if (inputFormat.getClass().getName().contains("HiveIcebergInputFormat")) {
+ boolean onlyOrcFiles = true;
+ Hive hiveDb = Hive.get(hiveConf, false);
+ for (String tableName : currentOperator.getSchema().getTableNames()) {
+ boolean isORCOnly =
+ "TRUE".equalsIgnoreCase(hiveDb.getTable(tableName).getParameters().get("orc.files.only"));
+ if (!isORCOnly) {
+ onlyOrcFiles = false;
Review Comment:
Yeps, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3607:
URL: https://github.com/apache/hive/pull/3607#discussion_r977158528
##########
iceberg/iceberg-handler/src/test/results/positive/alter_table_to_iceberg.q.out:
##########
@@ -103,6 +103,7 @@ Table Parameters:
metadata_location hdfs://### HDFS PATH ###
numFiles 1
numRows 5
+ orc.files.only false
Review Comment:
Sorted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1254504194
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3607: HIVE-26529: Iceberg: Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3607:
URL: https://github.com/apache/hive/pull/3607#issuecomment-1259285753
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3607)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3607&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3607&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3607&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org