You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ramesh0201 (via GitHub)" <gi...@apache.org> on 2024/12/12 17:17:47 UTC
[PR] HIVE-28408: Support ARRAY field access in CBO [hive]
ramesh0201 opened a new pull request, #5577:
URL: https://github.com/apache/hive/pull/5577
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
We need to support array field access in the CBO. This is currently already supported in the execution side. Only CBO fails with an error. So we will be by passing the CBO by doing no-op in this case to leave it to hive compiler and execution engine to do rest of the work
### Why are the changes needed?
In order to enable CBO for all the queries
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=nested_column_pruning.q
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2588135706
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1904225648
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
Review Comment:
The `Vectorizing complex type..` message is created by `getValidateDataTypeErrorMsg` during some validation type checks. Why aren't we hitting these checks before arriving into this method?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
+ }
StructTypeInfo structTypeInfo = (StructTypeInfo) structNodeDesc.getTypeInfo();
Review Comment:
The fact that we are hitting a `ClassCastException` without the extra if check above is a bit worrisome. It means that the new logic has some impact on the vectorization codepath. Why are we hitting this now?
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java:
##########
@@ -613,6 +617,14 @@ protected RexNode createConstantExpr(TypeInfo typeInfo, Object constantValue)
TypeConverter.convert(typeInfo, rexBuilder.getTypeFactory()), false);
}
+ /**
+ * Special operator that is used as syntactic sugar to change the type of collection
+ * expressions in order to perform field access over them.
+ */
+ public static final SqlOperator COMPONENT_ACCESS =
Review Comment:
Most of the existing Hive specific operators/functions are under `org.apache.hadoop.hive.ql.optimizer.calcite.reloperators` package (e.g., `HiveDateAddSqlOperator`). Consider moving the new operator into a separate class under that package.
Having one class per function is not ideal but it may be a good idea to follow the existing paradigm at least till we decide to perform a holistic refactoring that puts them all together.
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java:
##########
@@ -622,11 +634,13 @@ protected RexNode createNestedColumnRefExpr(
if (expr.getType().isStruct()) {
// regular case of accessing nested field in a column
return rexBuilder.makeFieldAccess(expr, fieldName, true);
+ } else if (expr.getType().getComponentType() != null) {
+ RexNode wrap =
+ rexBuilder.makeCall(expr.getType().getComponentType(), COMPONENT_ACCESS, Collections.singletonList(expr));
+ return createNestedColumnRefExpr(typeInfo, wrap, fieldName, isList);
} else {
- // This may happen for schema-less tables, where columns are dynamically
- // supplied by serdes.
- throw new CalciteSemanticException("Unexpected rexnode : "
- + expr.getClass().getCanonicalName(), UnsupportedFeature.Schema_less_table);
+ // Safe exception. Shouldn't Ideally come here.
Review Comment:
The comment does not provide much info; consider dropping it.
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java:
##########
@@ -622,11 +634,13 @@ protected RexNode createNestedColumnRefExpr(
if (expr.getType().isStruct()) {
// regular case of accessing nested field in a column
return rexBuilder.makeFieldAccess(expr, fieldName, true);
+ } else if (expr.getType().getComponentType() != null) {
+ RexNode wrap =
+ rexBuilder.makeCall(expr.getType().getComponentType(), COMPONENT_ACCESS, Collections.singletonList(expr));
+ return createNestedColumnRefExpr(typeInfo, wrap, fieldName, isList);
} else {
- // This may happen for schema-less tables, where columns are dynamically
- // supplied by serdes.
- throw new CalciteSemanticException("Unexpected rexnode : "
- + expr.getClass().getCanonicalName(), UnsupportedFeature.Schema_less_table);
Review Comment:
Can we remove the `UnsupportedFeature.Schema_less_table` from the enumeration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2590540439
Thank you for the review @okumin and @zabetak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2552109256
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1900634051
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -1867,7 +1867,7 @@ public static enum ConfVars {
+ "execution engine doesn't take advantage of statistics stored in the cache."),
// CBO related
HIVE_CBO_ENABLED("hive.cbo.enable", true, "Flag to control enabling Cost Based Optimizations using Calcite framework."),
- HIVE_CBO_FALLBACK_STRATEGY("hive.cbo.fallback.strategy", "CONSERVATIVE",
+ HIVE_CBO_FALLBACK_STRATEGY("hive.cbo.fallback.strategy", "NEVER",
Review Comment:
Yes I reverted this back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 merged PR #5577:
URL: https://github.com/apache/hive/pull/5577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1907500261
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
Review Comment:
We are hitting this message because we never do that check(call getValidateDataTypeErrorMsg) for the select clause expressions and do this check only for the group by expression, which is later in the operator tree for that particular vertex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2551350750
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2567457268
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1907506231
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
+ }
StructTypeInfo structTypeInfo = (StructTypeInfo) structNodeDesc.getTypeInfo();
Review Comment:
I was able to reproduce the ClassCastException even without my patch for a simpler query that I mentioned below:
EXPLAIN VECTORIZATION EXPRESSION
SELECT s5.f16.f18.f19 FROM nested_tbl_1;
I will be adding this to the list of queries in the same q file below. What I identified is that, we have certain nested access queries that were not handled well in the vectorization and we need to address that. But there is no new failures after our patch here. We can try to address this issue in a separate patch if you think it is reasonable idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1900633813
##########
ql/src/test/results/clientpositive/llap/vector_orc_nested_column_pruning.q.out:
##########
@@ -3008,7 +3008,7 @@ STAGE PLANS:
enabled: true
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- notVectorizedReason: Key expression for GROUPBY operator: Vectorizing complex type LIST not supported
+ notVectorizedReason: exception: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo stack trace: org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getStructFieldIndex(VectorizationContext.java:1106), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUDFStructField(VectorizationContext.java:1094), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:1074), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:972), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4806), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5443), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:1011), org.apache.hadoop.hive.ql
.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:898), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:868), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2500(Vectorizer.java:253), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2118), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2070), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:2045), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1199), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1051), ...
Review Comment:
Thank you @okumin. Please review my latest updated patch for this.
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java:
##########
@@ -622,11 +634,13 @@ protected RexNode createNestedColumnRefExpr(
if (expr.getType().isStruct()) {
// regular case of accessing nested field in a column
return rexBuilder.makeFieldAccess(expr, fieldName, true);
+ } else if (expr.getType().getComponentType() != null){
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2579370444
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1908747149
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
Review Comment:
I have not addressed this by calling the getValidateDataTypeErrorMsg() because, doing so will prevent any struct field in the select clause but right now we only fail if there is a list without an index in vectorization. I have also addressed the exception message accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2580500830
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2540204410
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1894873589
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -1867,7 +1867,7 @@ public static enum ConfVars {
+ "execution engine doesn't take advantage of statistics stored in the cache."),
// CBO related
HIVE_CBO_ENABLED("hive.cbo.enable", true, "Flag to control enabling Cost Based Optimizations using Calcite framework."),
- HIVE_CBO_FALLBACK_STRATEGY("hive.cbo.fallback.strategy", "CONSERVATIVE",
+ HIVE_CBO_FALLBACK_STRATEGY("hive.cbo.fallback.strategy", "NEVER",
Review Comment:
Can we make NEVER default? I guess this is the last known issue and that's why this PR would change it. It is one of the greatest news in 2024.
https://issues.apache.org/jira/browse/HIVE-27831
I'd create a separate ticket to change the default, cleaning up some test resources as [Stamatis tried before](https://github.com/apache/hive/pull/4836/files).
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/type/RexNodeExprFactory.java:
##########
@@ -622,11 +634,13 @@ protected RexNode createNestedColumnRefExpr(
if (expr.getType().isStruct()) {
// regular case of accessing nested field in a column
return rexBuilder.makeFieldAccess(expr, fieldName, true);
+ } else if (expr.getType().getComponentType() != null){
Review Comment:
I'd add a space after the right paren
##########
ql/src/test/results/clientpositive/llap/vector_orc_nested_column_pruning.q.out:
##########
@@ -3008,7 +3008,7 @@ STAGE PLANS:
enabled: true
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- notVectorizedReason: Key expression for GROUPBY operator: Vectorizing complex type LIST not supported
+ notVectorizedReason: exception: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo stack trace: org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getStructFieldIndex(VectorizationContext.java:1106), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUDFStructField(VectorizationContext.java:1094), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:1074), org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:972), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4806), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5443), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:1011), org.apache.hadoop.hive.ql
.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:898), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:868), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2500(Vectorizer.java:253), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2118), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2070), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:2045), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1199), org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1051), ...
Review Comment:
Although the result is unchanged, maybe we would like to have a better error here. ClassCastException sounds irregular. Also, this stack trace should not be robust
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2588612372
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2573266102
It would be also nice to include a qfile with an EXPLAIN CBO statement that shows that we can treat ARRAY field access in CBO. Possibly the book example that I added under HIVE-28408 is a good fit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1907506231
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
+ }
StructTypeInfo structTypeInfo = (StructTypeInfo) structNodeDesc.getTypeInfo();
Review Comment:
I was able to reproduce the ClassCastException even without the patch for a simpler query that I mentioned below:
EXPLAIN VECTORIZATION EXPRESSION
SELECT s5.f16.f18.f19 FROM nested_tbl_1;
I will be adding this to the list of queries in the same q file below. What I identified is that, we have certain nested access queries that were not handled well in the vectorization and we need to address that. But there is no new failures after our patch here. We can try to address this issue in a separate patch if you think it is reasonable idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5577:
URL: https://github.com/apache/hive/pull/5577#discussion_r1913857636
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
##########
@@ -1103,6 +1103,9 @@ private VectorExpression getGenericUDFStructField(ExprNodeFieldDesc exprNodeFiel
private int getStructFieldIndex(ExprNodeFieldDesc exprNodeFieldDesc) throws HiveException {
ExprNodeDesc structNodeDesc = exprNodeFieldDesc.getDesc();
String fieldName = exprNodeFieldDesc.getFieldName();
+ if (exprNodeFieldDesc.getIsList()) {
+ throw new HiveException("Could not vectorize expression: Vectorizing complex type LIST is not supported");
+ }
StructTypeInfo structTypeInfo = (StructTypeInfo) structNodeDesc.getTypeInfo();
Review Comment:
The new message is much better than getting a `ClassCastException`. Adding vectorized support for this path is outside the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2589470309
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577) **Quality Gate passed**
Issues
 [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5577&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5577&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5577&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5577)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28408: Support ARRAY field access in CBO [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on PR #5577:
URL: https://github.com/apache/hive/pull/5577#issuecomment-2579323672
@zabetak Can you please review the latest patch? I have addressed most of the comments, except the two that have I left a comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org