You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/18 13:55:04 UTC
[GitHub] [doris] qzsee opened a new pull request, #10981: [Enhancement](Nereids) add some basic data structure definitions.
qzsee opened a new pull request, #10981:
URL: https://github.com/apache/doris/pull/10981
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 merged pull request #10981: [Enhancement](Nereids) add some basic data structure definitions.
Posted by GitBox <gi...@apache.org>.
924060929 merged PR #10981:
URL: https://github.com/apache/doris/pull/10981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10981: [Enhancement](Nereids) add some basic data structure definitions.
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10981:
URL: https://github.com/apache/doris/pull/10981#discussion_r924041187
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -353,6 +384,31 @@ public UnboundFunction visitFunctionCall(DorisParser.FunctionCallContext ctx) {
});
}
+ @Override
+ public Expression visitInterval(IntervalContext ctx) {
+ return new IntervalLiteral((Expression) visit(ctx.value), (String) visit(ctx.unit));
Review Comment:
```suggestion
return new IntervalLiteral(getExpression(ctx.value), visitUnitIdentifier(ctx.unit));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -353,6 +384,31 @@ public UnboundFunction visitFunctionCall(DorisParser.FunctionCallContext ctx) {
});
}
+ @Override
+ public Expression visitInterval(IntervalContext ctx) {
+ return new IntervalLiteral((Expression) visit(ctx.value), (String) visit(ctx.unit));
+ }
+
+ @Override
+ public String visitUnitIdentifier(UnitIdentifierContext ctx) {
+ return ctx.getText();
+ }
+
+ @Override
+ public Expression visitTypeConstructor(TypeConstructorContext ctx) {
+ String value = ctx.STRING().getText();
+ value = value.substring(1, value.length() - 1);
+ String type = ctx.identifier().getText().toUpperCase();
+ switch (type) {
+ case "DATE":
+ return new DateLiteral(value, DateType.INSTANCE);
+ case "DATETIME":
+ return new DateLiteral(value, DateTimeType.INSTANCE);
Review Comment:
why use one class to represent two type?
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -220,6 +222,13 @@ booleanValue
: TRUE | FALSE
Review Comment:
this line is duplicated with `booleanValue`, please help to remove it, thx
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/IntervalLiteral.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+/**
+ * Interval for timestamp calculation.
+ */
+public class IntervalLiteral extends Expression {
+ private final Expression value;
+ private final String timeUnit;
Review Comment:
do we need add a unit enum?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java:
##########
@@ -83,6 +84,14 @@ public boolean isConstant() {
return children().stream().anyMatch(Expression::isConstant);
}
+ public final Expression castTo(DataType targetType) throws AnalysisException {
+ return uncheckedCastTo(targetType);
+ }
+
+ protected Expression uncheckedCastTo(DataType targetType) throws AnalysisException {
+ throw new RuntimeException();
Review Comment:
```suggestion
throw new RuntimeException("Do not implement uncheckedCastTo");
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindFunction.java:
##########
@@ -93,5 +104,25 @@ public Expression visitUnboundFunction(UnboundFunction unboundFunction, Void con
}
return unboundFunction;
}
+
+ @Override
+ public Expression visitTimestampArithmetic(TimestampArithmetic arithmetic, Void context) {
+ String funcOpName = null;
+ if (arithmetic.getFuncName() == null) {
+ funcOpName = String.format("%sS_%s", arithmetic.getTimeUnit(),
+ (arithmetic.getOp() == Operator.ADD) ? "ADD" : "SUB");
+ }
Review Comment:
do we need else
```java
else {
funcOpName = arithmetic.getFuncName();
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DateType.java:
##########
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.types;
+
+import org.apache.doris.catalog.Type;
+
+/**
+ * Date type in Nereids.
+ */
+public class DateType extends PrimitiveType {
+
+ public static DateType INSTANCE = new DateType();
+
+ @Override
+ public Type toCatalogDataType() {
+ return Type.DATE;
+ }
+
+ @Override
+ public boolean equals(Object o) {
Review Comment:
why need to override equals
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java:
##########
@@ -52,6 +52,12 @@ public static DataType convertFromCatalogDataType(Type catalogType) {
return VarcharType.createVarcharType(scalarType.getLength());
case STRING:
return StringType.INSTANCE;
+ case DATE:
+ return DateType.INSTANCE;
+ case DATETIME:
+ return DateTimeType.INSTANCE;
+ case DECIMALV2:
+ return DecimalType.INSTANCE;
Review Comment:
Decimal need to set precision and scale. BTW, we have DecimalV3 now. i.e. DECIMAL32, DECIMAL64 and DECIMAL128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #10981: [Enhancement](Nereids) add some basic data structure definitions.
Posted by GitBox <gi...@apache.org>.
morrySnow commented on PR #10981:
URL: https://github.com/apache/doris/pull/10981#issuecomment-1190544471
merge master code please~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #10981: [Enhancement](Nereids) add some basic data structure definitions.
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10981:
URL: https://github.com/apache/doris/pull/10981#discussion_r924606269
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/DateLiteral.java:
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DateTimeType;
+import org.apache.doris.nereids.types.DateType;
+import org.apache.doris.nereids.util.DateUtils;
+
+import com.google.common.base.Preconditions;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.joda.time.LocalDateTime;
+import org.joda.time.format.DateTimeFormatter;
+
+/**
+ * Date literal in Nereids.
+ */
+public class DateLiteral extends Literal {
+
+ private static final Logger LOG = LogManager.getLogger(DateLiteral.class);
+
+ private static final int DATEKEY_LENGTH = 8;
+ private static final int DATETIME_TO_MINUTE_STRING_LENGTH = 16;
+ private static final int DATETIME_TO_HOUR_STRING_LENGTH = 13;
+
+ private static DateTimeFormatter DATE_TIME_FORMATTER = null;
+ private static DateTimeFormatter DATE_TIME_FORMATTER_TO_HOUR = null;
+ private static DateTimeFormatter DATE_TIME_FORMATTER_TO_MINUTE = null;
+ private static DateTimeFormatter DATE_FORMATTER = null;
+ private static DateTimeFormatter DATE_TIME_FORMATTER_TWO_DIGIT = null;
+ private static DateTimeFormatter DATE_FORMATTER_TWO_DIGIT = null;
+ private static DateTimeFormatter DATEKEY_FORMATTER = null;
+
+ private long year;
+ private long month;
+ private long day;
+ private long hour;
+ private long minute;
+ private long second;
+ private long microsecond;
+
+ static {
+ try {
+ DATE_TIME_FORMATTER = DateUtils.formatBuilder("%Y-%m-%d %H:%i:%s").toFormatter();
+ DATE_TIME_FORMATTER_TO_HOUR = DateUtils.formatBuilder("%Y-%m-%d %H").toFormatter();
+ DATE_TIME_FORMATTER_TO_MINUTE = DateUtils.formatBuilder("%Y-%m-%d %H:%i").toFormatter();
+ DATE_FORMATTER = DateUtils.formatBuilder("%Y-%m-%d").toFormatter();
+ DATEKEY_FORMATTER = DateUtils.formatBuilder("%Y%m%d").toFormatter();
+ DATE_TIME_FORMATTER_TWO_DIGIT = DateUtils.formatBuilder("%y-%m-%d %H:%i:%s").toFormatter();
+ DATE_FORMATTER_TWO_DIGIT = DateUtils.formatBuilder("%y-%m-%d").toFormatter();
+ } catch (AnalysisException e) {
+ LOG.error("invalid date format", e);
+ System.exit(-1);
+ }
+ }
+
+ public DateLiteral(String s, DataType type) throws AnalysisException {
+ super(type);
+ init(s, type);
+ }
+
+ /**
+ * C'tor for date type.
+ */
+ public DateLiteral(long year, long month, long day) {
+ super(DateType.INSTANCE);
+ this.hour = 0;
+ this.minute = 0;
+ this.second = 0;
+ this.year = year;
+ this.month = month;
+ this.day = day;
+ }
+
+ /**
+ * C'tor for datetime type.
+ */
+ public DateLiteral(long year, long month, long day, long hour, long minute, long second) {
+ super(DateTimeType.INSTANCE);
+ this.hour = hour;
+ this.minute = minute;
+ this.second = second;
+ this.year = year;
+ this.month = month;
+ this.day = day;
+ }
+
+ /**
+ * C'tor for type conversion.
+ */
+ public DateLiteral(DateLiteral other, DataType type) {
+ super(type);
+ this.hour = other.hour;
+ this.minute = other.minute;
+ this.second = other.second;
+ this.year = other.year;
+ this.month = other.month;
+ this.day = other.day;
+ }
+
+ private void init(String s, DataType type) throws AnalysisException {
+ try {
+ LocalDateTime dateTime;
+ if (type.isDate()) {
+ if (s.split("-")[0].length() == 2) {
+ dateTime = DATE_FORMATTER_TWO_DIGIT.parseLocalDateTime(s);
+ } else if (s.length() == DATEKEY_LENGTH && !s.contains("-")) {
+ dateTime = DATEKEY_FORMATTER.parseLocalDateTime(s);
+ } else {
+ dateTime = DATE_FORMATTER.parseLocalDateTime(s);
+ }
+ } else {
+ if (s.split("-")[0].length() == 2) {
+ dateTime = DATE_TIME_FORMATTER_TWO_DIGIT.parseLocalDateTime(s);
+ } else {
+ if (s.length() == DATETIME_TO_MINUTE_STRING_LENGTH) {
+ dateTime = DATE_TIME_FORMATTER_TO_MINUTE.parseLocalDateTime(s);
+ } else if (s.length() == DATETIME_TO_HOUR_STRING_LENGTH) {
+ dateTime = DATE_TIME_FORMATTER_TO_HOUR.parseLocalDateTime(s);
+ } else {
+ dateTime = DATE_TIME_FORMATTER.parseLocalDateTime(s);
+ }
+ }
+ }
+ year = dateTime.getYear();
+ month = dateTime.getMonthOfYear();
+ day = dateTime.getDayOfMonth();
+ hour = dateTime.getHourOfDay();
+ minute = dateTime.getMinuteOfHour();
+ second = dateTime.getSecondOfMinute();
+ } catch (Exception ex) {
+ throw new AnalysisException("date literal [" + s + "] is invalid");
+ }
+ }
+
+ @Override
+ protected Expression uncheckedCastTo(DataType targetType) throws AnalysisException {
+ if (getDataType().equals(targetType)) {
+ return this;
+ }
+ if (targetType.isDateType()) {
+ if (getDataType().equals(targetType)) {
+ return this;
+ }
+ if (targetType.equals(DateType.INSTANCE)) {
+ return new DateLiteral(this.year, this.month, this.day);
+ } else if (targetType.equals(DateTimeType.INSTANCE)) {
+ return new DateLiteral(this.year, this.month, this.day, this.hour, this.minute, this.second);
+ } else {
+ throw new AnalysisException("Error date literal type : " + type);
+ }
+ }
+ //todo other target type cast
+ return this;
+ }
+
+ public DateLiteral withDataType(DataType type) {
+ Preconditions.checkArgument(type.isDate() || type.isDateTime());
+ return new DateLiteral(this, type);
+ }
+
+ @Override
+ public Long getValue() {
+ return (year * 10000 + month * 100 + day) * 1000000L + hour * 10000 + minute * 100 + second;
Review Comment:
we need to discuss later that should we return a human readable Long or a timestamp liked Long
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalType.java:
##########
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.types;
+
+import org.apache.doris.catalog.Type;
+
+/**
+ * Decimal type in Nereids.
+ */
+public class DecimalType extends FractionalType {
+
+ private int precision;
+ private int scale;
+
+ public DecimalType(int precision, int scale) {
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ public static DecimalType createDecimalType(int precision, int scale) {
+ return new DecimalType(precision, scale);
+ }
+
+ @Override
+ public Type toCatalogDataType() {
+ return Type.DECIMALV2;
Review Comment:
just like CatalogType to Type, we need to
```java
return ScalarType.createDecimalType(PrimitiveType.DECIMALV2, precision, scale);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/TimestampArithmetic.java:
##########
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.analysis.ArithmeticExpr.Operator;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+
+import com.google.common.base.Preconditions;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+
+/**
+ * Describes the addition and subtraction of time units from timestamps.
+ * Arithmetic expressions on timestamps are syntactic sugar.
+ * They are executed as function call exprs in the BE.
+ */
+public class TimestampArithmetic extends Expression implements BinaryExpression {
Review Comment:
just like Arithmetic, we need to different class to represent different TimestampArithmetic functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org