[GitHub] [flink] dawidwys commented on a change in pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …

[GitBox <gi...@apache.org>]

dawidwys commented on a change in pull request #13050:
URL: https://github.com/apache/flink/pull/13050#discussion_r470478732



##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java
##########
@@ -676,15 +680,20 @@ private Operation convertCreateView(SqlCreateView sqlCreateView) {
 		final SqlNode query = sqlCreateView.getQuery();
 		final SqlNodeList fieldList = sqlCreateView.getFieldList();
 
-		SqlNode validateQuery = flinkPlanner.validate(query);
 		// Put the sql string unparse (getQuotedSqlString()) in front of
 		// the node conversion (toQueryOperation()),
 		// because before Calcite 1.22.0, during sql-to-rel conversion, the SqlWindow
 		// bounds state would be mutated as default when they are null (not specified).
 
 		// This bug is fixed in CALCITE-3877 of Calcite 1.23.0.
 		String originalQuery = getQuotedSqlString(query);
-		String expandedQuery = getQuotedSqlString(validateQuery);
+		SqlNode validateQuery = flinkPlanner.validate(query);

Review comment:
       Can we extract the logic from alter view and create view to a common helper method? The code is nearly identical. I am pretty sure we can extract the majority of the logic to a method like: 
   ```
   CatalogView catalogView = transformToCatalogView(SqlNode query, String comment, List<String> aliases);
   ```
   
   Right now we need to modify the code in two different places and the comments differ significantly between the two locations, even though they should be exactly the same.

##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.utils;
+
+import org.apache.flink.table.planner.calcite.FlinkPlannerImpl;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlShuttle;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * Utility that expand SQL identifiers from a SQL query.

Review comment:
       ```suggestion
    * Utility that expands SQL identifiers in a SQL query.
   ```

##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.utils;
+
+import org.apache.flink.table.planner.calcite.FlinkPlannerImpl;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlShuttle;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * Utility that expand SQL identifiers from a SQL query.
+ *
+ * <p>Simple use:
+ *
+ * <blockquote><code>
+ * final String sql =<br>
+ *     "select ename from emp where deptno &lt; 10";<br>
+ * final Expander.Expanded expanded =<br>
+ *     Expander.create(planner).expanded(sql);<br>
+ * print(expanded); // "select `emp`.`ename` from `catalog`.`db`.`emp` where `emp`.`deptno` &lt; 10"
+ * </code></blockquote>
+ *
+ * <p>Calling {@link Expanded#toString()} generates a string that is similar to
+ * SQL where a user has manually converted all identifiers as expanded, and
+ * which could then be persisted as expanded query of a Catalog view.
+ *
+ * <p>For more advanced formatting, use {@link Expanded#substitute(Function)}.
+ *
+ * <p>Adjust {@link SqlParser.Config} to use a different parser or parsing options.
+ */
+public class Expander {
+	private final FlinkPlannerImpl planner;
+
+	private Expander(FlinkPlannerImpl planner) {
+		this.planner = Objects.requireNonNull(planner);
+	}
+
+	/** Creates an Expander. **/
+	public static Expander create(FlinkPlannerImpl planner) {
+		return new Expander(planner);
+	}
+
+	/** Expands identifiers in a given SQL string, returning a {@link Expanded}. */
+	public Expanded expanded(String ori) {
+		final Map<SqlParserPos, SqlIdentifier> identifiers = new HashMap<>();
+		final SqlNode oriNode;
+		final SqlNode validated;
+		oriNode = planner.parser().parse(ori);

Review comment:
       nit: inline assignment

##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.utils;
+
+import org.apache.flink.table.planner.calcite.FlinkPlannerImpl;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlShuttle;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * Utility that expand SQL identifiers from a SQL query.
+ *
+ * <p>Simple use:
+ *
+ * <blockquote><code>
+ * final String sql =<br>
+ *     "select ename from emp where deptno &lt; 10";<br>
+ * final Expander.Expanded expanded =<br>
+ *     Expander.create(planner).expanded(sql);<br>
+ * print(expanded); // "select `emp`.`ename` from `catalog`.`db`.`emp` where `emp`.`deptno` &lt; 10"
+ * </code></blockquote>
+ *
+ * <p>Calling {@link Expanded#toString()} generates a string that is similar to
+ * SQL where a user has manually converted all identifiers as expanded, and
+ * which could then be persisted as expanded query of a Catalog view.
+ *
+ * <p>For more advanced formatting, use {@link Expanded#substitute(Function)}.
+ *
+ * <p>Adjust {@link SqlParser.Config} to use a different parser or parsing options.
+ */
+public class Expander {

Review comment:
       Will it expand function identifiers as well? Can we have a test for that case?

##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.utils;
+
+import org.apache.flink.table.planner.calcite.FlinkPlannerImpl;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlShuttle;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * Utility that expand SQL identifiers from a SQL query.
+ *
+ * <p>Simple use:
+ *
+ * <blockquote><code>
+ * final String sql =<br>
+ *     "select ename from emp where deptno &lt; 10";<br>
+ * final Expander.Expanded expanded =<br>
+ *     Expander.create(planner).expanded(sql);<br>
+ * print(expanded); // "select `emp`.`ename` from `catalog`.`db`.`emp` where `emp`.`deptno` &lt; 10"
+ * </code></blockquote>
+ *
+ * <p>Calling {@link Expanded#toString()} generates a string that is similar to
+ * SQL where a user has manually converted all identifiers as expanded, and
+ * which could then be persisted as expanded query of a Catalog view.
+ *
+ * <p>For more advanced formatting, use {@link Expanded#substitute(Function)}.
+ *
+ * <p>Adjust {@link SqlParser.Config} to use a different parser or parsing options.
+ */
+public class Expander {
+	private final FlinkPlannerImpl planner;
+
+	private Expander(FlinkPlannerImpl planner) {
+		this.planner = Objects.requireNonNull(planner);
+	}
+
+	/** Creates an Expander. **/
+	public static Expander create(FlinkPlannerImpl planner) {
+		return new Expander(planner);
+	}
+
+	/** Expands identifiers in a given SQL string, returning a {@link Expanded}. */
+	public Expanded expanded(String ori) {
+		final Map<SqlParserPos, SqlIdentifier> identifiers = new HashMap<>();
+		final SqlNode oriNode;
+		final SqlNode validated;
+		oriNode = planner.parser().parse(ori);
+		// parse again because validation is stateful, that means the node tree was probably
+		// mutated.
+		validated = planner.validate(planner.parser().parse(ori));

Review comment:
       nit: inline assignment

##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/utils/Expander.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.utils;
+
+import org.apache.flink.table.planner.calcite.FlinkPlannerImpl;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.ImmutableMap;
+
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlShuttle;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+
+/**
+ * Utility that expand SQL identifiers from a SQL query.
+ *
+ * <p>Simple use:
+ *
+ * <blockquote><code>
+ * final String sql =<br>
+ *     "select ename from emp where deptno &lt; 10";<br>
+ * final Expander.Expanded expanded =<br>
+ *     Expander.create(planner).expanded(sql);<br>
+ * print(expanded); // "select `emp`.`ename` from `catalog`.`db`.`emp` where `emp`.`deptno` &lt; 10"
+ * </code></blockquote>
+ *
+ * <p>Calling {@link Expanded#toString()} generates a string that is similar to
+ * SQL where a user has manually converted all identifiers as expanded, and
+ * which could then be persisted as expanded query of a Catalog view.
+ *
+ * <p>For more advanced formatting, use {@link Expanded#substitute(Function)}.
+ *
+ * <p>Adjust {@link SqlParser.Config} to use a different parser or parsing options.
+ */
+public class Expander {
+	private final FlinkPlannerImpl planner;
+
+	private Expander(FlinkPlannerImpl planner) {
+		this.planner = Objects.requireNonNull(planner);
+	}
+
+	/** Creates an Expander. **/
+	public static Expander create(FlinkPlannerImpl planner) {
+		return new Expander(planner);
+	}
+
+	/** Expands identifiers in a given SQL string, returning a {@link Expanded}. */
+	public Expanded expanded(String ori) {
+		final Map<SqlParserPos, SqlIdentifier> identifiers = new HashMap<>();
+		final SqlNode oriNode;
+		final SqlNode validated;
+		oriNode = planner.parser().parse(ori);
+		// parse again because validation is stateful, that means the node tree was probably
+		// mutated.
+		validated = planner.validate(planner.parser().parse(ori));
+		validated.accept(new SqlBasicVisitor<Void>() {
+			@Override public Void visit(SqlIdentifier identifier) {
+				identifiers.putIfAbsent(identifier.getParserPosition(), identifier);
+				return null;
+			}
+		});
+		return new Expanded(planner.config().getParserConfig(), oriNode, identifiers);
+	}
+
+	/** Result of expanding. */
+	public static class Expanded {
+		public final SqlParser.Config parserConf;
+		public final SqlNode oriNode;
+		public final Map<SqlParserPos, SqlIdentifier> identifiersMap;
+
+		Expanded(SqlParser.Config parserConf, SqlNode oriNode,
+				Map<SqlParserPos, SqlIdentifier> identifiers) {
+			this.oriNode = oriNode;
+			this.parserConf = parserConf;
+			this.identifiersMap = ImmutableMap.copyOf(identifiers);
+		}
+
+		@Override
+		public String toString() {
+			return substitute(SqlNode::toString);
+		}
+
+		/** Returns the SQL string with identifiers replaced according to the
+		 * given unparse function. */
+		public String substitute(Function<SqlNode, String> fn) {
+			final SqlShuttle shuttle = new SqlShuttle() {
+				@Override
+				public SqlNode visit(SqlIdentifier id) {
+					if (id.isStar()) {
+						return id;
+					}
+					final SqlIdentifier toReplace = identifiersMap.get(id.getParserPosition());
+					if (toReplace != null && id.names.size() >= toReplace.names.size()) {

Review comment:
       Shouldn't this be rather:
   ```
   if (toReplace == null || id.names.size() >= toReplace.names.size()) {
   ```
   ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org