You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "kgyrtkirk (via GitHub)" <gi...@apache.org> on 2024/02/13 17:30:57 UTC
[PR] Table concat (druid)
kgyrtkirk opened a new pull request, #15897:
URL: https://github.com/apache/druid/pull/15897
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes #XXXX.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
#### Renamed the class ...
#### Added a forbidden-apis entry ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1503690679
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
Review Comment:
can this happen? isn't this first checked by the operand checker?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
Review Comment:
does this also take care of tables created via CTEs?
##########
sql/src/test/java/org/apache/druid/sql/calcite/CalciteTableAppendTest.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.sql.calcite.NotYetSupported.NotYetSupportedProcessor;
+import org.hamcrest.MatcherAssert;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class CalciteTableAppendTest extends BaseCalciteQueryTest
Review Comment:
is there a specific reason for leaving out the native query plans in the test queries?
##########
sql/src/test/java/org/apache/druid/sql/calcite/CalciteTableAppendTest.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.sql.calcite.NotYetSupported.NotYetSupportedProcessor;
+import org.hamcrest.MatcherAssert;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class CalciteTableAppendTest extends BaseCalciteQueryTest
+{
+ @Rule(order = 0)
+ public NotYetSupportedProcessor negativeTestProcessor = new NotYetSupportedProcessor();
+
+ @Test
+ public void testUnion()
+ {
+ testBuilder()
+ .sql("select dim1,null as dim4 from foo union all select dim1,dim4 from numfoo")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {"", null},
+ new Object[] {"10.1", null},
+ new Object[] {"2", null},
+ new Object[] {"1", null},
+ new Object[] {"def", null},
+ new Object[] {"abc", null},
+ new Object[] {"", "a"},
+ new Object[] {"10.1", "a"},
+ new Object[] {"2", "a"},
+ new Object[] {"1", "b"},
+ new Object[] {"def", "b"},
+ new Object[] {"abc", "b"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppend2()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo')) u")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"", null, null, null},
+ new Object[] {"10.1", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"1", null, null, null},
+ new Object[] {"def", null, null, null},
+ new Object[] {"abc", null, null, null},
+ new Object[] {"", "a", 1.0D, 1.0F},
+ new Object[] {"10.1", "a", 1.7D, 0.1F},
+ new Object[] {"2", "a", 0.0D, 0.0F},
+ new Object[] {"1", "b", null, null},
+ new Object[] {"def", "b", null, null},
+ new Object[] {"abc", "b", null, null}
+ )
+ )
+ .expectedLogicalPlan(
+ ""
+ + "LogicalProject(exprs=[[$1, $8, $11, $13]])\n"
+ + " LogicalTableScan(table=[[APPEND]])\n"
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendSameTableMultipleTimes()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo','foo')) u where dim1='2'")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", "a", 0.0D, 0.0F}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtSingleTableIsValid()
+ {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND('foo')) u")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {""},
+ new Object[] {"10.1"},
+ new Object[] {"2"},
+ new Object[] {"1"},
+ new Object[] {"def"},
+ new Object[] {"abc"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtNoTableIsInvalid()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND()) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs("No match found for function signature APPEND() (line [1], column [24])")
+ );
+ }
+ }
+
+ @Test
+ public void testAppendNoTableIsInvalid()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND()) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs("No match found for function signature APPEND() (line [1], column [24])")
+ );
+ }
+ }
+
+ @Test
+ public void testAppendtSingleTableIsInvalidArg()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND('foo',111)) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs(
+ "All arguments to APPEND should be literal strings. Argument #1 is not string (line [1], column [37])"
Review Comment:
do you think we should make the argument index printing to be 1 based instead of 0 based?
Also, I'm not sure whether it is possible to print the argument itself too, if we can do that then that'd be great too.
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
Review Comment:
is there a valid case with `null` table name?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
Review Comment:
the `SqlOperandMetadata` seems to say that it is only to be used for fixed number of parameters. do you think is it ok to use to use for variable params as well in this case?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ callBinding.operand(i)
+ );
+ }
+ tables.add(new TableOperand(callBinding.operand(i), table));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<TableOperand> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (TableOperand table : tables) {
+ RowSignature rowSignature = table.getRowSignature();
+ for (String columnName : rowSignature.getColumnNames()) {
+ ColumnType currentType = rowSignature.getColumnType(columnName).get();
+ ColumnType existingType = fields.get(columnName);
+
+ if (existingType == null || existingType.equals(currentType)) {
+ fields.put(columnName, currentType);
+ } else {
+ try {
+ ColumnType commonType = ColumnType.leastRestrictiveType(currentType, existingType);
+ fields.put(columnName, commonType);
+ }
+ catch (Exception e) {
Review Comment:
I think we should only catch `Types.IncompatibleTypeException` - is there a reason for a catch all?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ callBinding.operand(i)
+ );
+ }
+ tables.add(new TableOperand(callBinding.operand(i), table));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<TableOperand> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
Review Comment:
can move this to the actual place its being used
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
Review Comment:
should this generate a list of all the parameters with an independent name for each of them?
##########
sql/src/test/java/org/apache/druid/sql/calcite/CalciteTableAppendTest.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.sql.calcite.NotYetSupported.NotYetSupportedProcessor;
+import org.hamcrest.MatcherAssert;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class CalciteTableAppendTest extends BaseCalciteQueryTest
+{
+ @Rule(order = 0)
+ public NotYetSupportedProcessor negativeTestProcessor = new NotYetSupportedProcessor();
+
+ @Test
+ public void testUnion()
+ {
+ testBuilder()
+ .sql("select dim1,null as dim4 from foo union all select dim1,dim4 from numfoo")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {"", null},
+ new Object[] {"10.1", null},
+ new Object[] {"2", null},
+ new Object[] {"1", null},
+ new Object[] {"def", null},
+ new Object[] {"abc", null},
+ new Object[] {"", "a"},
+ new Object[] {"10.1", "a"},
+ new Object[] {"2", "a"},
+ new Object[] {"1", "b"},
+ new Object[] {"def", "b"},
+ new Object[] {"abc", "b"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppend2()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo')) u")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"", null, null, null},
+ new Object[] {"10.1", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"1", null, null, null},
+ new Object[] {"def", null, null, null},
+ new Object[] {"abc", null, null, null},
+ new Object[] {"", "a", 1.0D, 1.0F},
+ new Object[] {"10.1", "a", 1.7D, 0.1F},
+ new Object[] {"2", "a", 0.0D, 0.0F},
+ new Object[] {"1", "b", null, null},
+ new Object[] {"def", "b", null, null},
+ new Object[] {"abc", "b", null, null}
+ )
+ )
+ .expectedLogicalPlan(
+ ""
+ + "LogicalProject(exprs=[[$1, $8, $11, $13]])\n"
+ + " LogicalTableScan(table=[[APPEND]])\n"
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendSameTableMultipleTimes()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo','foo')) u where dim1='2'")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", "a", 0.0D, 0.0F}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtSingleTableIsValid()
+ {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND('foo')) u")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {""},
+ new Object[] {"10.1"},
+ new Object[] {"2"},
+ new Object[] {"1"},
+ new Object[] {"def"},
+ new Object[] {"abc"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtNoTableIsInvalid()
Review Comment:
seems like a redundant test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1505761377
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
I've moved the section to be the next after `union`
what do you think:
* should this be made a subsection of `union`?
* should I remove the `native` as that matches the standard union's behaviour?
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
referenced updated the query and referenced [table-level union](sql.md#table-level) as that's more closer to what the user might be working with
thank you!
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
I've added a backreferencing paragraph to that as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Table concat (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1488345810
##########
processing/src/main/java/org/apache/druid/segment/column/RowSignature.java:
##########
@@ -373,6 +373,15 @@
{
return new RowSignature(columnTypeList);
}
+
+ public void putAll(RowSignature rowSignature)
Review Comment:
## Useless parameter
The parameter 'rowSignature' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/6561)
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/ConcatTableMacro.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.schema.FunctionParameter;
+import org.apache.calcite.schema.TableMacro;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.schema.impl.ReflectiveFunctionBase;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * FIXME
+ */
+public class ConcatTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+ public ConcatTableMacro(SqlOperandMetadata t)
+ {
+ super(
+ new SqlIdentifier(TableConcatOperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ t,
+ null
+ );
+
+ // this.macro = new MyTableMacro();
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("t1", "t2").build();
+ }
+
+ static class MyTableMacro implements TableMacro
+ {
+
+ @Override
+ public List<FunctionParameter> getParameters()
+ {
+ final ReflectiveFunctionBase.ParameterListBuilder params = ReflectiveFunctionBase.builder();
+
+ params.add(String.class, "T1");
+ params.add(String.class, "T2");
+ return params.build();
+
+ }
+
+ @Override
+ public TranslatableTable apply(List<? extends @Nullable Object> arguments)
+ {
+ if (true) {
+ throw new RuntimeException("FIXME: Unimplemented!");
+ }
+ return null;
+ }
+
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlCallBinding ss = (SqlCallBinding) callBinding;
+ SqlValidator validator = ss.getValidator();
+ SqlValidatorCatalogReader catalogReader = validator.getCatalogReader();
+ Object t = catalogReader.getTable(ImmutableList.<String>builder().add("foo").build());
Review Comment:
## Unread local variable
Variable 'Object t' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/6566)
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/ConcatTableMacro.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.schema.FunctionParameter;
+import org.apache.calcite.schema.TableMacro;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.schema.impl.ReflectiveFunctionBase;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * FIXME
+ */
+public class ConcatTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+ public ConcatTableMacro(SqlOperandMetadata t)
+ {
+ super(
+ new SqlIdentifier(TableConcatOperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ t,
+ null
+ );
+
+ // this.macro = new MyTableMacro();
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("t1", "t2").build();
+ }
+
+ static class MyTableMacro implements TableMacro
+ {
+
+ @Override
+ public List<FunctionParameter> getParameters()
+ {
+ final ReflectiveFunctionBase.ParameterListBuilder params = ReflectiveFunctionBase.builder();
+
+ params.add(String.class, "T1");
+ params.add(String.class, "T2");
+ return params.build();
+
+ }
+
+ @Override
+ public TranslatableTable apply(List<? extends @Nullable Object> arguments)
+ {
+ if (true) {
+ throw new RuntimeException("FIXME: Unimplemented!");
+ }
+ return null;
+ }
+
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlCallBinding ss = (SqlCallBinding) callBinding;
+ SqlValidator validator = ss.getValidator();
+ SqlValidatorCatalogReader catalogReader = validator.getCatalogReader();
+ Object t = catalogReader.getTable(ImmutableList.<String>builder().add("foo").build());
+
+ List<String> tableNames = getTableNames(callBinding);
+ List<RelOptTable> tables = getTables(catalogReader, tableNames);
+
+ AppendDesc union = buildUnionDataSource(tables);
+ return new AppendTable(union);
+ }
+
+ static class AppendDesc
+ {
+ private RowSignature values;
+ private List<DataSource> dataSources;
+
+ public AppendDesc(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSources = dataSources;
+ }
+
+ public DatasourceMetadata buildPhysicalDatasourceMetadata()
+ {
+ return new DatasourceMetadata()
+ {
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return new UnionDataSource(dataSources);
+ }
+ };
+ }
+ }
+
+ private AppendDesc buildUnionDataSource(List<RelOptTable> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, RelDataTypeField> fields = new LinkedHashMap<>();
+ Map<String, ColumnType> fields2 = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (RelOptTable relOptTable : tables) {
+
+ if (false) {
+ for (RelDataTypeField newField : relOptTable.getRowType().getFieldList()) {
+
+ String key = newField.getKey();
+ RelDataTypeField existingField = fields.get(key);
+ if (existingField != null && !existingField.equals(newField)) {
+ // FIXME this could be more sophisticated
+ throw new IllegalArgumentException("incompatible operands");
+ }
+ if (existingField == null) {
+ fields.put(key, newField);
+ }
+ }
+ }
+ DatasourceTable a = relOptTable.unwrapOrThrow(DatasourceTable.class);
+
+ RowSignature rowSignature = a.getRowSignature();
+ for (String currentColumn : rowSignature.getColumnNames()) {
+
+ String key = currentColumn;
+ ColumnType currentType = rowSignature.getColumnType(currentColumn).get();
+ ColumnType existingType = fields2.get(key);
+ if (existingType != null && !existingType.equals(currentType)) {
+ // FIXME this could be more sophisticated
+ throw new IllegalArgumentException("incompatible operands");
+ }
+ if (existingType == null) {
+ fields2.put(key, currentType);
+ }
+ }
+
+ dataSources.add(a.getDataSource());
+ }
+
+ for (Entry<String, ColumnType> col : fields2.entrySet()) {
+ rowSignatureBuilder.add(col.getKey(), col.getValue());
+ }
+ return new AppendDesc(rowSignatureBuilder.build(), dataSources);
+ }
+
+ private List<String> getTableNames(SqlOperatorBinding callBinding)
+ {
+ List<String> ret = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
+ }
+ ret.add(callBinding.getOperandLiteralValue(i, String.class));
+ }
+ return ret;
+ }
+
+ private List<RelOptTable> getTables(SqlValidatorCatalogReader catalogReader, List<String> tableNames)
+ {
+ List<RelOptTable> ret = new ArrayList<>();
+ for (String tableName : tableNames) {
+ SqlValidatorTable t = catalogReader.getTable(ImmutableList.<String>builder().add(tableName).build());
+ ret.add(t.unwrapOrThrow(RelOptTable.class));
+ }
+ return ret;
+ }
+
+ private TranslatableTable apply(List<RelOptTable> tables)
Review Comment:
## Useless parameter
The parameter 'tables' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/6562)
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/ConcatTableMacro.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.schema.FunctionParameter;
+import org.apache.calcite.schema.TableMacro;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.schema.impl.ReflectiveFunctionBase;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * FIXME
+ */
+public class ConcatTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+ public ConcatTableMacro(SqlOperandMetadata t)
+ {
+ super(
+ new SqlIdentifier(TableConcatOperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ t,
+ null
+ );
+
+ // this.macro = new MyTableMacro();
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("t1", "t2").build();
+ }
+
+ static class MyTableMacro implements TableMacro
Review Comment:
## Unused classes and interfaces
Unused class: MyTableMacro is not referenced within this codebase. If not used as an external API it should be removed.
[Show more details](https://github.com/apache/druid/security/code-scanning/6567)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg merged PR #15897:
URL: https://github.com/apache/druid/pull/15897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "imply-cheddar (via GitHub)" <gi...@apache.org>.
imply-cheddar commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1498759731
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlValidator validator = ((SqlCallBinding) callBinding).getValidator();
+ List<RelOptTable> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ private List<RelOptTable> getTables(SqlOperatorBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<RelOptTable> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ ((SqlCallBinding) callBinding).operand(i)
+ );
+ }
+ tables.add(table.unwrapOrThrow(RelOptTable.class));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<RelOptTable> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (RelOptTable relOptTable : tables) {
+ DatasourceTable table = relOptTable.unwrapOrThrow(DatasourceTable.class);
+ RowSignature rowSignature = table.getRowSignature();
+ for (String columnName : rowSignature.getColumnNames()) {
+ ColumnType currentType = rowSignature.getColumnType(columnName).get();
+ ColumnType existingType = fields.get(columnName);
+
+ if (existingType == null || existingType.equals(currentType)) {
+ fields.put(columnName, currentType);
+ } else {
+ try {
+ ColumnType commonType = ColumnType.leastRestrictiveType(currentType, existingType);
+ fields.put(columnName, commonType);
+ }
+ catch (Exception e) {
+ throw InvalidInput.exception(
Review Comment:
`InvalidSqlInput`?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlValidator validator = ((SqlCallBinding) callBinding).getValidator();
+ List<RelOptTable> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ private List<RelOptTable> getTables(SqlOperatorBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<RelOptTable> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ ((SqlCallBinding) callBinding).operand(i)
+ );
+ }
+ tables.add(table.unwrapOrThrow(RelOptTable.class));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<RelOptTable> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (RelOptTable relOptTable : tables) {
+ DatasourceTable table = relOptTable.unwrapOrThrow(DatasourceTable.class);
+ RowSignature rowSignature = table.getRowSignature();
+ for (String columnName : rowSignature.getColumnNames()) {
+ ColumnType currentType = rowSignature.getColumnType(columnName).get();
+ ColumnType existingType = fields.get(columnName);
+
+ if (existingType == null || existingType.equals(currentType)) {
+ fields.put(columnName, currentType);
+ } else {
+ try {
+ ColumnType commonType = ColumnType.leastRestrictiveType(currentType, existingType);
+ fields.put(columnName, commonType);
+ }
+ catch (Exception e) {
+ throw InvalidInput.exception(
Review Comment:
If we don't except this Exception to ever happen, then maybe make it a `DruidException.defensive()` instead and perhaps add something to the message about why it should never occur.
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlValidator validator = ((SqlCallBinding) callBinding).getValidator();
+ List<RelOptTable> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ private List<RelOptTable> getTables(SqlOperatorBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<RelOptTable> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
Review Comment:
Are we certain that this error message will make it back to the user? I'm not sure what phase we are in here. If we want this message to get to the user, I wonder about making it an `InvalidSqlInput.exception()` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1505079620
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
* Since this has the same native query syntax as `union` should this content go within the existing `union` section?
* Suggest removing the SQL syntax from the header title, so just have `#### Dynamic table append`
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
+
+This is essentially an SQL level syntactic sugar which matches columns by name from multiple tables.
Review Comment:
```suggestion
Perform dynamic table appends in SQL using `TABLE(APPEND(...))`. This simplifies SQL syntax to match columns by name from multiple tables. The native query syntax remains the same as for native union datasources.
```
##########
docs/querying/datasource.md:
##########
@@ -177,7 +177,15 @@ use union datasources.
### `inline`
<Tabs>
-<TabItem value="7" label="Native">
+<TabItem value="sql" label="SQL">
Review Comment:
Thanks for adding this missing example
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
+
+This is essentially an SQL level syntactic sugar which matches columns by name from multiple tables.
+Suppose you have 3 tables:
+* `table1` has `column1`
+* `table2` has `column2`
+* `table3` has `column1`, `column2`, `column3`
+
+It is possible to create an union view of the above tables is possible by using UNION ALL:
Review Comment:
```suggestion
You can create a union view of all the tables by using UNION ALL:
```
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
+
+This is essentially an SQL level syntactic sugar which matches columns by name from multiple tables.
+Suppose you have 3 tables:
+* `table1` has `column1`
+* `table2` has `column2`
+* `table3` has `column1`, `column2`, `column3`
+
+It is possible to create an union view of the above tables is possible by using UNION ALL:
+```sql
+SELECT column1,NULL AS column2,NULL AS column3 FROM table1
+UNION ALL
+SELECT NULL AS column1,column2,NULL AS column3 FROM table2
+UNION ALL
+SELECT column1,column2,column3 FROM table3
+```
+
+However depending on the size of the table's schema it might be quite complicated to do that; `TABLE(APPEND('table1','table2','table3'))` represents the same in a more compact form.
+
+<Tabs>
Review Comment:
All the sections that have these tabs do so at the beginning of the section, so consider moving this to the top for consistency
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
Consider also including a line / link to this content from the [SQL syntax doc](https://druid.apache.org/docs/latest/querying/sql#union-all) in https://github.com/apache/druid/blob/master/docs/querying/sql.md?plain=1#L322
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
+
+This is essentially an SQL level syntactic sugar which matches columns by name from multiple tables.
+Suppose you have 3 tables:
Review Comment:
```suggestion
Suppose you have three tables:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1491016435
##########
sql/src/main/java/org/apache/druid/sql/calcite/planner/DruidSqlValidator.java:
##########
@@ -177,7 +177,7 @@ public void validateCall(SqlCall call, SqlValidatorScope scope)
super.validateCall(call, scope);
}
- private CalciteContextException buildCalciteContextException(String message, SqlNode call)
+ public static CalciteContextException buildCalciteContextException(String message, SqlNode call)
Review Comment:
better place?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
Review Comment:
changed it to `"tableName1", "tableName2"`
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlValidator validator = ((SqlCallBinding) callBinding).getValidator();
+ List<RelOptTable> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ private List<RelOptTable> getTables(SqlOperatorBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<RelOptTable> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
Review Comment:
didn't notice this remained an `IllegalArgumentException` ; fixed it!
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlValidator validator = ((SqlCallBinding) callBinding).getValidator();
+ List<RelOptTable> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ private List<RelOptTable> getTables(SqlOperatorBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<RelOptTable> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw new IllegalArgumentException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal"
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ ((SqlCallBinding) callBinding).operand(i)
+ );
+ }
+ tables.add(table.unwrapOrThrow(RelOptTable.class));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<RelOptTable> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (RelOptTable relOptTable : tables) {
+ DatasourceTable table = relOptTable.unwrapOrThrow(DatasourceTable.class);
+ RowSignature rowSignature = table.getRowSignature();
+ for (String columnName : rowSignature.getColumnNames()) {
+ ColumnType currentType = rowSignature.getColumnType(columnName).get();
+ ColumnType existingType = fields.get(columnName);
+
+ if (existingType == null || existingType.equals(currentType)) {
+ fields.put(columnName, currentType);
+ } else {
+ try {
+ ColumnType commonType = ColumnType.leastRestrictiveType(currentType, existingType);
+ fields.put(columnName, commonType);
+ }
+ catch (Exception e) {
+ throw InvalidInput.exception(
Review Comment:
It doesn't seem like `ColumnType.leastRestrictiveType` will throw exception right now - but that may change in the futue...better to handle it correctly
changed it to use `buildCalciteContextException`
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
Review Comment:
I had to do some back-and-forth before implementing these things; mostly because (as you've noticed) originally the tablemacro supposed to have a fixed number of arguments.
I don't really understand why it needs to have fixed number of arguments...
however I wanted to use [this logic](https://github.com/apache/calcite/blob/2558c13cdb8b6a8e1608112a902b1bf9d97b5386/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L2874) - but for that I had to retain the `SqlUserDefinedTableMacro` type; which wants `SqlOperandMetadata` which have the idea of fixed number of arguments....
implementing `SqlOperandMetadata` made it work correctly - but `paramNames` and `paramTypes` remained odd...
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
Review Comment:
no; `null` is not a string literal either ; so it can't happen => but added a testcase ; in case that changes
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
Review Comment:
right - this is not necessary; removed it!
##########
sql/src/test/java/org/apache/druid/sql/calcite/CalciteTableAppendTest.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.sql.calcite.NotYetSupported.NotYetSupportedProcessor;
+import org.hamcrest.MatcherAssert;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class CalciteTableAppendTest extends BaseCalciteQueryTest
Review Comment:
added native queries
##########
sql/src/main/java/org/apache/druid/sql/calcite/table/DatasourceMetadata.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.table;
+
+import org.apache.druid.query.DataSource;
+
+public interface DatasourceMetadata
+{
+
+ DataSource dataSource();
Review Comment:
apidoc?
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/AppendTableMacro.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class AppendTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new AppendTableMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private AppendTableMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ RelDataType type = callBinding.getOperandType(i);
+
+ SqlTypeName typeName = type.getSqlTypeName();
+
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(2);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>, <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding callBinding)
+ {
+ SqlCallBinding ss = (SqlCallBinding) callBinding;
+ SqlValidator validator = ss.getValidator();
+ SqlValidatorCatalogReader catalogReader = validator.getCatalogReader();
+ List<String> tableNames = getTableNames(callBinding);
+ List<RelOptTable> tables = getTables(catalogReader, tableNames);
+
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<RelOptTable> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (RelOptTable relOptTable : tables) {
+
+ DatasourceTable a = relOptTable.unwrapOrThrow(DatasourceTable.class);
Review Comment:
rename
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ callBinding.operand(i)
+ );
+ }
+ tables.add(new TableOperand(callBinding.operand(i), table));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<TableOperand> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
+ for (TableOperand table : tables) {
+ RowSignature rowSignature = table.getRowSignature();
+ for (String columnName : rowSignature.getColumnNames()) {
+ ColumnType currentType = rowSignature.getColumnType(columnName).get();
+ ColumnType existingType = fields.get(columnName);
+
+ if (existingType == null || existingType.equals(currentType)) {
+ fields.put(columnName, currentType);
+ } else {
+ try {
+ ColumnType commonType = ColumnType.leastRestrictiveType(currentType, existingType);
+ fields.put(columnName, commonType);
+ }
+ catch (Exception e) {
Review Comment:
don't think anything else might be coming out; changed it
##########
sql/src/test/java/org/apache/druid/sql/calcite/CalciteTableAppendTest.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.sql.calcite.NotYetSupported.NotYetSupportedProcessor;
+import org.hamcrest.MatcherAssert;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class CalciteTableAppendTest extends BaseCalciteQueryTest
+{
+ @Rule(order = 0)
+ public NotYetSupportedProcessor negativeTestProcessor = new NotYetSupportedProcessor();
+
+ @Test
+ public void testUnion()
+ {
+ testBuilder()
+ .sql("select dim1,null as dim4 from foo union all select dim1,dim4 from numfoo")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {"", null},
+ new Object[] {"10.1", null},
+ new Object[] {"2", null},
+ new Object[] {"1", null},
+ new Object[] {"def", null},
+ new Object[] {"abc", null},
+ new Object[] {"", "a"},
+ new Object[] {"10.1", "a"},
+ new Object[] {"2", "a"},
+ new Object[] {"1", "b"},
+ new Object[] {"def", "b"},
+ new Object[] {"abc", "b"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppend2()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo')) u")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"", null, null, null},
+ new Object[] {"10.1", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"1", null, null, null},
+ new Object[] {"def", null, null, null},
+ new Object[] {"abc", null, null, null},
+ new Object[] {"", "a", 1.0D, 1.0F},
+ new Object[] {"10.1", "a", 1.7D, 0.1F},
+ new Object[] {"2", "a", 0.0D, 0.0F},
+ new Object[] {"1", "b", null, null},
+ new Object[] {"def", "b", null, null},
+ new Object[] {"abc", "b", null, null}
+ )
+ )
+ .expectedLogicalPlan(
+ ""
+ + "LogicalProject(exprs=[[$1, $8, $11, $13]])\n"
+ + " LogicalTableScan(table=[[APPEND]])\n"
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendSameTableMultipleTimes()
+ {
+ testBuilder()
+ .sql("select dim1,dim4,d1,f1 from TABLE(APPEND('foo','numfoo','foo')) u where dim1='2'")
+ .expectedResults(
+ ResultMatchMode.RELAX_NULLS,
+ ImmutableList.of(
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", null, null, null},
+ new Object[] {"2", "a", 0.0D, 0.0F}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtSingleTableIsValid()
+ {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND('foo')) u")
+ .expectedResults(
+ ImmutableList.of(
+ new Object[] {""},
+ new Object[] {"10.1"},
+ new Object[] {"2"},
+ new Object[] {"1"},
+ new Object[] {"def"},
+ new Object[] {"abc"}
+ )
+ )
+ .run();
+ }
+
+ @Test
+ public void testAppendtNoTableIsInvalid()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND()) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs("No match found for function signature APPEND() (line [1], column [24])")
+ );
+ }
+ }
+
+ @Test
+ public void testAppendNoTableIsInvalid()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND()) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs("No match found for function signature APPEND() (line [1], column [24])")
+ );
+ }
+ }
+
+ @Test
+ public void testAppendtSingleTableIsInvalidArg()
+ {
+ try {
+ testBuilder()
+ .sql("select dim1 from TABLE(APPEND('foo',111)) u")
+ .run();
+ Assert.fail("query execution should fail");
+ }
+ catch (DruidException e) {
+ MatcherAssert.assertThat(
+ e,
+ invalidSqlIs(
+ "All arguments to APPEND should be literal strings. Argument #1 is not string (line [1], column [37])"
Review Comment:
totally agree - changed it!
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
Review Comment:
no - or not yet; `ProcedureNamespace` is deriving the type for the arguments; so can't use plain identifiers =>
with string arguments I was not able to make it work
I don't know if its possible to do that right now - but it would be nice to have that; most likely the direct identifier approach will be the best.
Anyway - if its a CTE; then there will be real subquery inputs; which wouldn't be easy to handle right now....or I just don't see a way to make that work
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
+ if (table == null) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ StringUtils.format("Table [%s] not found", tableName),
+ callBinding.operand(i)
+ );
+ }
+ tables.add(new TableOperand(callBinding.operand(i), table));
+ }
+ return tables;
+ }
+
+ static class AppendDatasourceMetadata implements DatasourceMetadata
+ {
+ private final RowSignature values;
+ private final DataSource dataSource;
+
+ public AppendDatasourceMetadata(RowSignature values, List<DataSource> dataSources)
+ {
+ this.values = values;
+ this.dataSource = new UnionDataSource(dataSources);
+ }
+
+ @Override
+ public boolean isJoinable()
+ {
+ return false;
+ }
+
+ @Override
+ public boolean isBroadcast()
+ {
+ return false;
+ }
+
+ @Override
+ public DataSource dataSource()
+ {
+ return dataSource;
+ }
+ }
+
+ private AppendDatasourceMetadata buildUnionDataSource(List<TableOperand> tables)
+ {
+ List<DataSource> dataSources = new ArrayList<>();
+ Map<String, ColumnType> fields = new LinkedHashMap<>();
+ Builder rowSignatureBuilder = RowSignature.builder();
Review Comment:
oh - right! I've extracted it into a method as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1505765390
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
Review Comment:
Thanks for the explanation - it makes sense to only support persisted tables for now. This can be considered when the execution engine supports CTE + table union easily.
We can mention this caveat in the documentation for the users though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1505815820
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+
+ SqlTypeName typeName = callBinding.getOperandType(i).getSqlTypeName();
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings. "
+ + "Argument #" + i + " is not string",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public SqlOperandCountRange getOperandCountRange()
+ {
+ return SqlOperandCountRanges.from(1);
+ }
+
+ @Override
+ public String getAllowedSignatures(SqlOperator op, String opName)
+ {
+ return "APPEND( <TABLE_NAME>[, <TABLE_NAME> ...] )";
+ }
+
+ @Override
+ public List<RelDataType> paramTypes(RelDataTypeFactory typeFactory)
+ {
+ RelDataType t = typeFactory.createSqlType(SqlTypeName.VARCHAR);
+ return ImmutableList.<RelDataType>builder().add(t, t).build();
+ }
+
+ @Override
+ public List<String> paramNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+ }
+
+ @Override
+ public List<String> getParamNames()
+ {
+ return ImmutableList.<String>builder().add("tableName", "tableName").build();
+ }
+
+ @Override
+ public TranslatableTable getTable(SqlOperatorBinding operatorBinding)
+ {
+ SqlCallBinding callBinding = (SqlCallBinding) operatorBinding;
+ SqlValidator validator = callBinding.getValidator();
+
+ List<TableOperand> tables = getTables(callBinding, validator.getCatalogReader());
+ AppendDatasourceMetadata metadata = buildUnionDataSource(tables);
+ return new DatasourceTable(
+ metadata.values,
+ metadata,
+ EffectiveMetadata.of(metadata.values)
+ );
+ }
+
+ static class TableOperand
+ {
+ private final SqlNode sqlOperand;
+ private final SqlValidatorTable table;
+
+ public TableOperand(SqlNode sqlOperand, SqlValidatorTable table)
+ {
+ this.sqlOperand = sqlOperand;
+ this.table = table;
+ }
+
+ public RelOptTable getRelOptTable()
+ {
+ return table.unwrapOrThrow(RelOptTable.class);
+ }
+
+ public DatasourceTable getDataSourceTable()
+ {
+ return table.unwrapOrThrow(DatasourceTable.class);
+ }
+
+ public RowSignature getRowSignature()
+ {
+ return getDataSourceTable().getRowSignature();
+ }
+
+ public DataSource getDataSource()
+ {
+ return getDataSourceTable().getDataSource();
+ }
+ }
+
+ private List<TableOperand> getTables(SqlCallBinding callBinding, SqlValidatorCatalogReader catalogReader)
+ {
+ List<TableOperand> tables = new ArrayList<>();
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ if (!callBinding.isOperandLiteral(i, false)) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments of call to macro "
+ + "APPEND should be literal. Actual argument #"
+ + i + " is not literal",
+ callBinding.operand(i)
+ );
+ }
+ @Nullable
+ String tableName = callBinding.getOperandLiteralValue(i, String.class);
+ ImmutableList<String> tableNameList = ImmutableList.<String>builder().add(tableName).build();
+ SqlValidatorTable table = catalogReader.getTable(tableNameList);
Review Comment:
added the following `info` to the docs:
> [!NOTE]
> Only tables defined in the catalog are supported in `TABLE(APPEND())` - due to that; common table expressions result in `table not found` errors for queries like:
> ```sql
> WITH cte_table AS (SELECT * from TABLE1) SELECT * FROM TABLE(APPEND('cte_table'))
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1506409515
##########
docs/querying/datasource.md:
##########
@@ -211,6 +219,50 @@ Inline datasources are not available in Druid SQL.
Refer to the [Query execution](query-execution.md#inline) page for more details on how queries are executed when you
use inline datasources.
+### Dynamic table append `TABLE(APPEND(...))`
Review Comment:
I like both of those ideas. Since the native union behavior is the same, the new content can go in there, and removing the repeated `native` section won't have people wondering what the difference is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "vtlim (via GitHub)" <gi...@apache.org>.
vtlim commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1506418329
##########
docs/querying/datasource.md:
##########
@@ -174,15 +174,14 @@ table unions.
Refer to the [Query execution](query-execution.md#union) page for more details on how queries are executed when you
use union datasources.
-### `inline`
+### `dynamic table append`
Review Comment:
Since this isn't a property value, we don't need code font.
```suggestion
### Dynamic table append
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Introduce dynamic table append (druid)
Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1505766972
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/TableAppendMacro.java:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class TableAppendMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new TableAppendMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private TableAppendMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
Review Comment:
discussed offline, the approach taken seems like the only decent way to use `SqlUserDefinedTableMacro` - so we can keep using it. In future, we may want to add support in calcite itself for variadic params in `SqlOperandMetadata` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Table concat (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15897:
URL: https://github.com/apache/druid/pull/15897#discussion_r1489542493
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/AppendTableMacro.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class AppendTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new AppendTableMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private AppendTableMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings."
+ + "Argument #" + i + " is not literal",
+ operand
+ );
+ } else {
+ return false;
+ }
+ }
+ RelDataType type = callBinding.getOperandType(i);
+
+ SqlTypeName typeName = type.getSqlTypeName();
+
+ if (!SqlTypeFamily.CHARACTER.getTypeNames().contains(typeName)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings."
+ + "Argument #" + i + " is not string",
Review Comment:
## Missing space in string literal
This string appears to be missing a space after 'strings.'.
[Show more details](https://github.com/apache/druid/security/code-scanning/6570)
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/AppendTableMacro.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.external;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.TranslatableTable;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlOperandMetadata;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorCatalogReader;
+import org.apache.calcite.sql.validate.SqlValidatorTable;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.UnionDataSource;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.column.RowSignature.Builder;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
+import org.apache.druid.sql.calcite.planner.DruidSqlValidator;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.DatasourceMetadata;
+import org.apache.druid.sql.calcite.table.DatasourceTable;
+import org.apache.druid.sql.calcite.table.DatasourceTable.EffectiveMetadata;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+/**
+ * Dynamic table append.
+ *
+ * Enables to use: TABLE(APPEND('t1','t2')); which will provide a union view of the operand tables.
+ */
+public class AppendTableMacro extends SqlUserDefinedTableMacro implements AuthorizableOperator
+{
+
+ public static final OperatorConversion OPERATOR_CONVERSION = new OperatorConversion();
+ public static final SqlOperator APPEND_TABLE_MACRO = new AppendTableMacro();
+
+ private static class OperatorConversion implements SqlOperatorConversion
+ {
+ public static final String FUNCTION_NAME = "APPEND";
+
+ public OperatorConversion()
+ {
+ }
+
+ @Override
+ public SqlOperator calciteOperator()
+ {
+ return APPEND_TABLE_MACRO;
+ }
+
+ @Override
+ public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
+ {
+ throw new IllegalStateException();
+ }
+ }
+
+ private AppendTableMacro()
+ {
+ super(
+ new SqlIdentifier(OperatorConversion.FUNCTION_NAME, SqlParserPos.ZERO),
+ SqlKind.OTHER_FUNCTION,
+ ReturnTypes.CURSOR,
+ null,
+ new OperandMetadata(),
+ null
+ );
+ }
+
+ private static class OperandMetadata implements SqlOperandMetadata
+ {
+ @Override
+ public boolean checkOperandTypes(SqlCallBinding callBinding, boolean throwOnFailure)
+ {
+ for (int i = 0; i < callBinding.getOperandCount(); i++) {
+ SqlNode operand = callBinding.operand(i);
+ if (!callBinding.isOperandLiteral(i, false)) {
+ if (throwOnFailure) {
+ throw DruidSqlValidator.buildCalciteContextException(
+ "All arguments to APPEND should be literal strings."
+ + "Argument #" + i + " is not literal",
Review Comment:
## Missing space in string literal
This string appears to be missing a space after 'strings.'.
[Show more details](https://github.com/apache/druid/security/code-scanning/6569)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org