You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2023/01/04 07:36:05 UTC
[GitHub] [flink] lsyldliu opened a new pull request, #21596: [FLINK-29719][hive] Supports native count function for hive dialect
lsyldliu opened a new pull request, #21596:
URL: https://github.com/apache/flink/pull/21596
## What is the purpose of the change
*Implementing a native hive count function based on the expression aggregate function, can improve the agg performance by using hash-agg strategy.*
## Brief change log
- *Supports native count function for hive dialect*
## Verifying this change
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests in HiveDialectQueryITCase*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
lsyldliu commented on PR #21596:
URL: https://github.com/apache/flink/pull/21596#issuecomment-1378134535
cc @beyond1920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
luoyuxia commented on code in PR #21596:
URL: https://github.com/apache/flink/pull/21596#discussion_r1069363291
##########
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/functions/hive/HiveCountAggFunction.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.functions.hive;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.flink.table.expressions.UnresolvedReferenceExpression;
+import org.apache.flink.table.planner.expressions.ExpressionBuilder;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.inference.CallContext;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.table.expressions.ApiExpressionUtils.unresolvedRef;
+import static org.apache.flink.table.planner.expressions.ExpressionBuilder.ifThenElse;
+import static org.apache.flink.table.planner.expressions.ExpressionBuilder.literal;
+import static org.apache.flink.table.planner.expressions.ExpressionBuilder.plus;
+
+/** built-in hive count aggregate function. */
+public class HiveCountAggFunction extends HiveDeclarativeAggregateFunction {
+
+ private final UnresolvedReferenceExpression count = unresolvedRef("count");
+ private Integer arguments;
+ private boolean count1;
+
+ @Override
+ public int operandCount() {
+ return arguments;
+ }
+
+ @Override
+ public UnresolvedReferenceExpression[] aggBufferAttributes() {
+ return new UnresolvedReferenceExpression[] {count};
+ }
+
+ @Override
+ public DataType[] getAggBufferTypes() {
+ return new DataType[] {DataTypes.BIGINT()};
+ }
+
+ @Override
+ public DataType getResultType() {
+ return DataTypes.BIGINT();
+ }
+
+ @Override
+ public Expression[] initialValuesExpressions() {
+ return new Expression[] {/* count = */ literal(0L, getResultType().notNull())};
+ }
+
+ @Override
+ public Expression[] accumulateExpressions() {
+ // count(*) and count(1) mean that count all elements
+ if (arguments == 0 || count1) {
+ return new Expression[] {/* count = */ plus(count, literal(1L))};
+ }
+
+ // other case need to determine the value of the element
+ List<Expression> operandExpressions = new ArrayList<>();
+ for (int i = 0; i < arguments; i++) {
+ operandExpressions.add(operand(i));
+ }
+ Expression operandExpression =
+ operandExpressions.stream()
+ .map(ExpressionBuilder::isNull)
+ .reduce(ExpressionBuilder::or)
+ .get();
+ return new Expression[] {
+ /* count = */ ifThenElse(operandExpression, count, plus(count, literal(1L)))
+ };
+ }
+
+ @Override
+ public Expression[] retractExpressions() {
+ throw new TableException("Count aggregate function does not support retraction.");
+ }
+
+ @Override
+ public Expression[] mergeExpressions() {
+ return new Expression[] {/* count = */ plus(count, mergeOperand(count))};
+ }
+
+ @Override
+ public Expression getValueExpression() {
+ return count;
+ }
+
+ @Override
+ public void setArguments(CallContext callContext) {
+ if (arguments == null) {
+ arguments = callContext.getArgumentDataTypes().size();
+ if (arguments == 1) {
+ // If the argument is literal 1 indicates use count(1)
Review Comment:
It may be `count(2)`,`count(3)`..., not always 1. May be we can optimize the comment as follows?
```suggestion
// If the argument is literal, indicates use count(literal)
```
If so, the varaible may also need changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe closed pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
godfreyhe closed pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
URL: https://github.com/apache/flink/pull/21596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
lsyldliu commented on PR #21596:
URL: https://github.com/apache/flink/pull/21596#issuecomment-1377196295
cc @luoyuxia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21596:
URL: https://github.com/apache/flink/pull/21596#issuecomment-1370577686
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ae33e5d7d7916ef45ba2189ae5a2e16a911425f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae33e5d7d7916ef45ba2189ae5a2e16a911425f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae33e5d7d7916ef45ba2189ae5a2e16a911425f2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
lsyldliu commented on PR #21596:
URL: https://github.com/apache/flink/pull/21596#issuecomment-1386433018
@luoyuxia have rebased the master and improved the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #21596: [FLINK-29719][hive] Supports native count function for hive dialect
Posted by GitBox <gi...@apache.org>.
lsyldliu commented on PR #21596:
URL: https://github.com/apache/flink/pull/21596#issuecomment-1383119790
@luoyuxia Thanks for reviewing, I've addressed the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org