You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by GitBox <gi...@apache.org> on 2023/01/11 07:34:11 UTC
[GitHub] [calcite] wnob opened a new pull request, #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
wnob opened a new pull request, #3023:
URL: https://github.com/apache/calcite/pull/3023
This incorporates the type alias work drafted by Julian in https://github.com/julianhyde/calcite/tree/5424-custom-literals and builds on it by incorporating my own first draft to address CALCITE-5180 ([Implement BigQuery date/time constructors](https://github.com/apache/calcite/commit/eb281b1706a6435f791b886ea86dc2aa65faf0f7)), and then adding in the `DATETIME()` function making use of the type aliases ([Add DATETIME() function and switch to the core parser](https://github.com/apache/calcite/commit/6e5b2f22414b1c1617d0e721d6dcf93f1b82d673)), and also switches the parsing logic from the Babel parser to the core parser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] olivrlee commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
olivrlee commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1072911432
##########
babel/src/test/resources/sql/big-query.iq:
##########
@@ -1097,12 +1097,74 @@ select unix_date(datetime '2008-12-25') as d;
#
# Return Data Type: DATE
-select date(2022, 11, 15) as d;
-+------------+
-| d |
-+------------+
-| 2022-11-15 |
-+------------+
+select date(2022, 11, 15) as d1,
+ date(datetime "2008-01-01 01:03:05") as d2,
+ date(datetime(2008, 1, 1, 1, 3, 5)) as d3;
++------------+------------+------------+
+| d1 | d2 | d3 |
++------------+------------+------------+
+| 2022-11-15 | 2008-01-01 | 2008-01-01 |
++------------+------------+------------+
+(1 row)
+
+!ok
+
+# Test timezone conversion when converting TIMESTAMP to DATE.
+# Denver observes DST whereas Phoenix does not.
+# Both cities have a -07:00 offset in winter, but Denver has -06:00 in summer.
+select date(timestamp("2008-06-21 06:30:00")) as sum_utc,
+ date(timestamp("2008-06-21 06:30:00"), "America/Denver") as sum_dst,
+ date(timestamp("2008-06-21 06:30:00"), "America/Phoenix") as sum_std,
+ date(timestamp("2008-12-21 06:30:00")) as win_utc,
+ date(timestamp("2008-12-21 06:30:00"), "America/Denver") as win_dst,
+ date(timestamp("2008-12-21 06:30:00"), "America/Phoenix") as win_std;
++------------+------------+------------+------------+------------+------------+
+| sum_utc | sum_dst | sum_std | win_utc | win_dst | win_std |
++------------+------------+------------+------------+------------+------------+
+| 2008-06-21 | 2008-06-21 | 2008-06-20 | 2008-12-21 | 2008-12-20 | 2008-12-20 |
++------------+------------+------------+------------+------------+------------+
+(1 row)
+
+!ok
+
+#####################################################################
+# DATETIME
Review Comment:
there's a set of DATETIME tests currently disabled around line 650 btw
https://github.com/apache/calcite/blob/6e5b2f22414b1c1617d0e721d6dcf93f1b82d673/babel/src/test/resources/sql/big-query.iq#L649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] wnob commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
wnob commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1074030646
##########
core/src/main/java/org/apache/calcite/sql/SqlUnknownLiteral.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql;
+
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.parser.SqlParserUtil;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.NlsString;
+import org.apache.calcite.util.Util;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Literal whose type is not yet known.
+ */
+public class SqlUnknownLiteral extends SqlLiteral {
+ public final String tag;
+
+ SqlUnknownLiteral(String tag, String value, SqlParserPos pos) {
+ super(requireNonNull(value, "value"), SqlTypeName.UNKNOWN, pos);
+ this.tag = requireNonNull(tag, "tag");
+ }
+
+ @Override public String getValue() {
+ return (String) requireNonNull(super.getValue(), "value");
+ }
+
+ @Override public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
+ final NlsString nlsString = new NlsString(getValue(), null, null);
+ writer.keyword(tag);
+ writer.literal(nlsString.asSql(true, true, writer.getDialect()));
+ }
+
+
+ /** Converts this unknown literal to a literal of known type. */
+ public SqlLiteral resolve(SqlTypeName typeName) {
+ switch (typeName) {
+ case DATE:
+ return SqlParserUtil.parseDateLiteral(getValue(), pos);
+ case TIME:
+ return SqlParserUtil.parseTimeLiteral(getValue(), pos);
+ case TIMESTAMP:
+ return SqlParserUtil.parseTimestampLiteral(getValue(), pos);
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return SqlParserUtil.parseTimestampWithLocalTimeZoneLiteral(getValue(), pos);
+ default:
Review Comment:
See in `SqlValidatorImpl.resolveLiteral` (which calls this function):
```
final SqlIdentifier identifier =
new SqlIdentifier(unknownLiteral.tag, SqlParserPos.ZERO);
final @Nullable RelDataType type = catalogReader.getNamedType(identifier);
final SqlTypeName typeName;
if (type != null) {
typeName = type.getSqlTypeName();
} else {
typeName = SqlTypeName.lookup(unknownLiteral.tag);
}
return unknownLiteral.resolve(typeName);
```
So, it makes sense to call `SqlLiteral.createUnknown("DATETIME")`, passing "DATETIME" as the `tag`. The tag just says "this is what the type called itself", and in order to resolve it, we look up in `catalogReader` to see if it's mapped to anything. In the case of BQ, it will map to `TIMESTAMP`. In the case of everything else, it would be null, so validation would fail when it invokes `SqlTypeName.lookup(unknownLiteral.tag)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] olivrlee commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
olivrlee commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1069975023
##########
core/src/main/java/org/apache/calcite/sql/SqlUnknownLiteral.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql;
+
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.parser.SqlParserUtil;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.NlsString;
+import org.apache.calcite.util.Util;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Literal whose type is not yet known.
+ */
+public class SqlUnknownLiteral extends SqlLiteral {
+ public final String tag;
+
+ SqlUnknownLiteral(String tag, String value, SqlParserPos pos) {
+ super(requireNonNull(value, "value"), SqlTypeName.UNKNOWN, pos);
+ this.tag = requireNonNull(tag, "tag");
+ }
+
+ @Override public String getValue() {
+ return (String) requireNonNull(super.getValue(), "value");
+ }
+
+ @Override public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
+ final NlsString nlsString = new NlsString(getValue(), null, null);
+ writer.keyword(tag);
+ writer.literal(nlsString.asSql(true, true, writer.getDialect()));
+ }
+
+
+ /** Converts this unknown literal to a literal of known type. */
+ public SqlLiteral resolve(SqlTypeName typeName) {
+ switch (typeName) {
+ case DATE:
+ return SqlParserUtil.parseDateLiteral(getValue(), pos);
+ case TIME:
+ return SqlParserUtil.parseTimeLiteral(getValue(), pos);
+ case TIMESTAMP:
+ return SqlParserUtil.parseTimestampLiteral(getValue(), pos);
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return SqlParserUtil.parseTimestampWithLocalTimeZoneLiteral(getValue(), pos);
+ default:
Review Comment:
did you miss a case for DATETIME?
https://github.com/apache/calcite/pull/3023/files#diff-e873041549333502af52ece8a1b34301ae5a059ff4719e9bddbaef48929e7047R4617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] asfgit closed pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
URL: https://github.com/apache/calcite/pull/3023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] olivrlee commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
olivrlee commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1070115107
##########
core/src/main/java/org/apache/calcite/sql/SqlUnknownLiteral.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql;
+
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.parser.SqlParserUtil;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.NlsString;
+import org.apache.calcite.util.Util;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Literal whose type is not yet known.
+ */
+public class SqlUnknownLiteral extends SqlLiteral {
+ public final String tag;
+
+ SqlUnknownLiteral(String tag, String value, SqlParserPos pos) {
+ super(requireNonNull(value, "value"), SqlTypeName.UNKNOWN, pos);
+ this.tag = requireNonNull(tag, "tag");
+ }
+
+ @Override public String getValue() {
+ return (String) requireNonNull(super.getValue(), "value");
+ }
+
+ @Override public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
+ final NlsString nlsString = new NlsString(getValue(), null, null);
+ writer.keyword(tag);
+ writer.literal(nlsString.asSql(true, true, writer.getDialect()));
+ }
+
+
+ /** Converts this unknown literal to a literal of known type. */
+ public SqlLiteral resolve(SqlTypeName typeName) {
+ switch (typeName) {
+ case DATE:
+ return SqlParserUtil.parseDateLiteral(getValue(), pos);
+ case TIME:
+ return SqlParserUtil.parseTimeLiteral(getValue(), pos);
+ case TIMESTAMP:
+ return SqlParserUtil.parseTimestampLiteral(getValue(), pos);
+ case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+ return SqlParserUtil.parseTimestampWithLocalTimeZoneLiteral(getValue(), pos);
+ default:
Review Comment:
nvm discussed offline, I think that location should be `return SqlLiteral.createUnknown("TIMESTAMP")` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] olivrlee commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
olivrlee commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1070146181
##########
core/src/main/codegen/templates/Parser.jj:
##########
@@ -4609,15 +4610,116 @@ SqlLiteral DateTimeLiteral() :
}
|
<DATE> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseDateLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("DATE", p, s.end(this));
+ }
+|
+ <DATETIME> { s = span(); } p = SimpleStringLiteral() {
+ return SqlLiteral.createUnknown("DATETIME", p, s.end(this));
}
|
<TIME> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseTimeLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("TIME", p, s.end(this));
}
|
+ LOOKAHEAD(2)
<TIMESTAMP> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseTimestampLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("TIMESTAMP", p, s.end(this));
+ }
+|
+ <TIMESTAMP> { s = span(); } <WITH> <LOCAL> <TIME> <ZONE> p = SimpleStringLiteral() {
+ return SqlLiteral.createUnknown("TIMESTAMP WITH LOCAL TIME ZONE", p, s.end(this));
+ }
+}
+
+/**
+ * Parses BigQuery's built-in DATE() function.
+ */
+SqlNode DateFunctionCall() :
+{
+ final SqlFunctionCategory funcType = SqlFunctionCategory.TIMEDATE;
+ final SqlIdentifier qualifiedName;
+ final Span s;
+ final SqlLiteral quantifier;
+ final List<? extends SqlNode> args;
+}
+{
+ <DATE> {
+ s = span();
+ qualifiedName = new SqlIdentifier(unquotedIdentifier(), getPos());
+ }
+ args = FunctionParameterList(ExprContext.ACCEPT_SUB_QUERY) {
+ quantifier = (SqlLiteral) args.get(0);
+ args.remove(0);
Review Comment:
what is this line doing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
julianhyde commented on PR #3023:
URL: https://github.com/apache/calcite/pull/3023#issuecomment-1384675118
There's too much going on here for me to review.
Can you separate it out into pieces that can be understood by an end user as adding some useful functionality? If there refactorings that don't change functionality, keep those separate.
It's confusing that there are type aliases, and there are also functions that are named after types. Basically you need to craft commit messages that are suitable for the release notes. (And each such commit should be a jira case that explains the new functionality.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] wnob commented on a diff in pull request #3023: [CALCITE-5180] Implement BigQuery Date/Time Type Aliases and Constructors
Posted by GitBox <gi...@apache.org>.
wnob commented on code in PR #3023:
URL: https://github.com/apache/calcite/pull/3023#discussion_r1074027016
##########
core/src/main/codegen/templates/Parser.jj:
##########
@@ -4609,15 +4610,116 @@ SqlLiteral DateTimeLiteral() :
}
|
<DATE> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseDateLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("DATE", p, s.end(this));
+ }
+|
+ <DATETIME> { s = span(); } p = SimpleStringLiteral() {
+ return SqlLiteral.createUnknown("DATETIME", p, s.end(this));
}
|
<TIME> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseTimeLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("TIME", p, s.end(this));
}
|
+ LOOKAHEAD(2)
<TIMESTAMP> { s = span(); } p = SimpleStringLiteral() {
- return SqlParserUtil.parseTimestampLiteral(p, s.end(this));
+ return SqlLiteral.createUnknown("TIMESTAMP", p, s.end(this));
+ }
+|
+ <TIMESTAMP> { s = span(); } <WITH> <LOCAL> <TIME> <ZONE> p = SimpleStringLiteral() {
+ return SqlLiteral.createUnknown("TIMESTAMP WITH LOCAL TIME ZONE", p, s.end(this));
+ }
+}
+
+/**
+ * Parses BigQuery's built-in DATE() function.
+ */
+SqlNode DateFunctionCall() :
+{
+ final SqlFunctionCategory funcType = SqlFunctionCategory.TIMEDATE;
+ final SqlIdentifier qualifiedName;
+ final Span s;
+ final SqlLiteral quantifier;
+ final List<? extends SqlNode> args;
+}
+{
+ <DATE> {
+ s = span();
+ qualifiedName = new SqlIdentifier(unquotedIdentifier(), getPos());
+ }
+ args = FunctionParameterList(ExprContext.ACCEPT_SUB_QUERY) {
+ quantifier = (SqlLiteral) args.get(0);
+ args.remove(0);
Review Comment:
If you look at the production for `FunctionParameterList()` you'll see it start like this:
```
{
<LPAREN>
(
qualifier = AllOrDistinct() { list.add(qualifier); }
|
{ list.add(null); }
)
...
```
Here, `list` is the return value (`args` here in `DateFunctionCall()`). So the first argument is always either `null`, or either `ALL` or `DISTINCT`, since some functions can take those qualifiers. Here, it should always be `null`, although I suppose this means the parser would silently accept something like `DATE(DISTINCT 2023, 01, 18)`, which it probably shouldn't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org