You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/03/21 16:29:30 UTC
[GitHub] [hive] kasakrisz opened a new pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
kasakrisz opened a new pull request #3121:
URL: https://github.com/apache/hive/pull/3121
### What changes were proposed in this pull request?
Add nullability to `ColumnInfo` and use it when creating `RexNodes`
### Why are the changes needed?
When TableScan logical operators are created table constraint information is pulled from the metastore and applied the RelDataType associated to the TS operator.
Constraints like not null has an affect the way Hive rewrites subqueries to joins. Example: if the optimizer is aware that the lhs of an in subquery can not be null a simpler rewrite is chosen. See jira for example plans.
### Does this PR introduce _any_ user-facing change?
No, but `explain` command output may be different in older Hive versions.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestTezTPCDS30TBPerfCliDriver -Dqfile=query45.q -pl itests/qtest -Pitests
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=subquery_in.q -pl itests/qtest -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz merged pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
Posted by GitBox <gi...@apache.org>.
kasakrisz merged pull request #3121:
URL: https://github.com/apache/hive/pull/3121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a change in pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
Posted by GitBox <gi...@apache.org>.
zabetak commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832062572
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
- for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
- }
- }
- if (nnPos == -1) {
- LOG.error("Column for not null constraint definition " + nnCol + " not found");
- return rowType;
- }
- builder.set(nnPos);
- }
- }
- if (pkc != null) {
- for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
- }
- }
- if (pkPos == -1) {
- LOG.error("Column for not null constraint definition " + pkCol + " not found");
- return rowType;
- }
- builder.set(pkPos);
- }
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
+ notNullConstraints.getNotNullConstraints().containsValue(colName)) {
+ return false;
}
- ImmutableBitSet bitSet = builder.build();
-
- RexBuilder rexBuilder = cluster.getRexBuilder();
- RelDataTypeFactory dtFactory = rexBuilder.getTypeFactory();
- List<RelDataType> fieldTypes = new LinkedList<RelDataType>();
- List<String> fieldNames = new LinkedList<String>();
- for (RelDataTypeField rdtf : rowType.getFieldList()) {
- if (bitSet.indexOf(rdtf.getIndex()) != -1) {
- fieldTypes.add(dtFactory.createTypeWithNullability(rdtf.getType(), false));
- } else {
- fieldTypes.add(rdtf.getType());
- }
- fieldNames.add(rdtf.getName());
+ if (primaryKeyInfo != null &&
+ !primaryKeyInfo.getColNames().isEmpty() &&
Review comment:
Do we need the `isEmpty` check?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
- for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
- }
- }
- if (nnPos == -1) {
- LOG.error("Column for not null constraint definition " + nnCol + " not found");
- return rowType;
- }
- builder.set(nnPos);
- }
- }
- if (pkc != null) {
- for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
- }
- }
- if (pkPos == -1) {
- LOG.error("Column for not null constraint definition " + pkCol + " not found");
- return rowType;
- }
- builder.set(pkPos);
- }
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
Review comment:
Do we need the `isEmpty` check?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
Review comment:
Do we need this `if` block? If both are null we are going to reach the final `return true` statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a change in pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832123342
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
- for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
- }
- }
- if (nnPos == -1) {
- LOG.error("Column for not null constraint definition " + nnCol + " not found");
- return rowType;
- }
- builder.set(nnPos);
- }
- }
- if (pkc != null) {
- for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
- }
- }
- if (pkPos == -1) {
- LOG.error("Column for not null constraint definition " + pkCol + " not found");
- return rowType;
- }
- builder.set(pkPos);
- }
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
Review comment:
Removed
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
- for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
- }
- }
- if (nnPos == -1) {
- LOG.error("Column for not null constraint definition " + nnCol + " not found");
- return rowType;
- }
- builder.set(nnPos);
- }
- }
- if (pkc != null) {
- for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
- if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
- }
- }
- if (pkPos == -1) {
- LOG.error("Column for not null constraint definition " + pkCol + " not found");
- return rowType;
- }
- builder.set(pkPos);
- }
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
+ notNullConstraints.getNotNullConstraints().containsValue(colName)) {
+ return false;
}
- ImmutableBitSet bitSet = builder.build();
-
- RexBuilder rexBuilder = cluster.getRexBuilder();
- RelDataTypeFactory dtFactory = rexBuilder.getTypeFactory();
- List<RelDataType> fieldTypes = new LinkedList<RelDataType>();
- List<String> fieldNames = new LinkedList<String>();
- for (RelDataTypeField rdtf : rowType.getFieldList()) {
- if (bitSet.indexOf(rdtf.getIndex()) != -1) {
- fieldTypes.add(dtFactory.createTypeWithNullability(rdtf.getType(), false));
- } else {
- fieldTypes.add(rdtf.getType());
- }
- fieldNames.add(rdtf.getName());
+ if (primaryKeyInfo != null &&
+ !primaryKeyInfo.getColNames().isEmpty() &&
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on pull request #3121:
URL: https://github.com/apache/hive/pull/3121#issuecomment-1075127798
Partition columns can have not null constraint defined so I extended this patch to handle them.
Virtual columns can have null values. Ex.:
```
SELECT ROW__ID FROM non_transactional;
```
will produce `NULLs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a change in pull request #3121: HIVE-26043: Use constraint info when creating RexNodes
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832123658
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias, QB qb) throws SemanticExc
return tableRel;
}
- private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType rowType)
- throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
- return rowType;
+ private boolean isNullable(String colName, NotNullConstraint notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+ return true;
}
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org