You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2023/01/04 04:58:26 UTC
[GitHub] [druid] paul-rogers opened a new pull request, #13627: Much improved table functions
paul-rogers opened a new pull request, #13627:
URL: https://github.com/apache/druid/pull/13627
### Description
Another step toward the Druid catalog, this one providing extended functionality to the table functions introduced in an earlier PR.
* Revises properties, definitions in the catalog
* Adds a "table function" abstraction to model such functions
* Specific functions for HTTP, inline, local and S3.
* Extended SQL types in the catalog. Primarily for later use, also used to handle types for external tables.
* Restructure external table definitions to use table functions
* EXTEND syntax for Druid's extern table function
* Support for array-valued table function parameters
* Support for array-valued SQL query parameters
Prior to this PR, MSQ did not support query parameters. This PR starts to provide support. Some of the required functionality will follow in the planner-focused catalog PR.
#### Release note
See the `reference.md` documentation page in this PR for the revised functionality. Of note are the new table functions and the array table function parameters. This functionality allows the user to, say, reuse the same SQL for every ingest, only passing in a different set of input files as query parameters on each ingest.
<hr>
This PR has:
- [X] been self-reviewed.
- [X] added documentation for new or modified features or behaviors.
- [X] a release note entry in the PR description.
- [X] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [X] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [X] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066344426
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -93,15 +100,54 @@ public static <T> T safeCast(Object value, Class<T> type, String propertyName)
catch (ClassCastException e) {
throw new IAE("Value [%s] is not valid for property %s, expected type %s",
value,
- propertyName,
+ key,
type.getSimpleName()
);
}
}
- public static <T> T safeGet(Map<String, Object> map, String propertyName, Class<T> type)
+ public static <T> T safeGet(Map<String, Object> map, String key, Class<T> type)
{
- return safeCast(map.get(propertyName), type, propertyName);
+ return safeCast(map.get(key), type, key);
+ }
+
+ public static long getLong(Map<String, Object> map, String key)
+ {
+ Object value = map.get(key);
+ if (value == null) {
+ return 0;
+ }
+
+ // Jackson may deserialize the value as either Integer or Long.
+ if (value instanceof Integer) {
+ return (Integer) value;
+ }
+ return (Long) value;
+ }
+
+ public static String getString(Map<String, Object> map, String key)
+ {
+ return safeGet(map, key, String.class);
+ }
+
+ public static List<String> getStringList(Map<String, Object> map, String key)
Review Comment:
this isnt used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066588944
##########
server/src/main/java/org/apache/druid/catalog/model/table/DatasourceDefn.java:
##########
@@ -19,65 +19,137 @@
package org.apache.druid.catalog.model.table;
+import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
import org.apache.druid.catalog.model.ColumnSpec;
import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.GranularityPropertyDefn;
+import org.apache.druid.catalog.model.ModelProperties.StringListPropertyDefn;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefn;
import org.apache.druid.java.util.common.IAE;
import org.apache.druid.java.util.common.StringUtils;
-import java.util.Collections;
+import java.util.Arrays;
+import java.util.List;
-/**
- * Definition of a Druid datasource. The datasource may use rollup, but rollup
- * is not represented in the catalog: it is just a way that applications store
- * data into a table.
- */
-public class DatasourceDefn extends AbstractDatasourceDefn
+public class DatasourceDefn extends TableDefn
{
/**
- * Definition of a column in a datasource.
+ * Segment grain at ingestion and initial compaction. Aging rules
+ * may override the value as segments age. If not provided here,
+ * then it must be provided at ingestion time.
*/
- public static class DatasourceColumnDefn extends ColumnDefn
- {
- public static final String COLUMN_TYPE = "column";
+ public static final String SEGMENT_GRANULARITY_PROPERTY = "segmentGranularity";
+
+ /**
+ * Catalog property value for the "all time" granularity.
+ */
+ public static final String ALL_GRANULARITY = "ALL";
Review Comment:
What is this a separate property from the segment granularity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1062019970
##########
server/src/test/java/org/apache/druid/catalog/model/table/DatasourceTableTest.java:
##########
@@ -195,107 +206,172 @@
@Test
public void testColumnSpec()
{
- // Type is required
- {
- ColumnSpec spec = new ColumnSpec(null, null, null, null);
- assertThrows(IAE.class, () -> spec.validate());
- }
-
// Name is required
{
- ColumnSpec spec = new ColumnSpec(DatasourceColumnDefn.COLUMN_TYPE, null, null, null);
+ ColumnSpec spec = new ColumnSpec(null, null, null);
assertThrows(IAE.class, () -> spec.validate());
}
{
- ColumnSpec spec = new ColumnSpec(DatasourceColumnDefn.COLUMN_TYPE, "foo", null, null);
+ ColumnSpec spec = new ColumnSpec("foo", null, null);
spec.validate();
}
// Type is optional
{
- ColumnSpec spec = new ColumnSpec(DatasourceColumnDefn.COLUMN_TYPE, "foo", "VARCHAR", null);
+ ColumnSpec spec = new ColumnSpec("foo", "VARCHAR", null);
spec.validate();
}
}
@Test
- public void testDetailTableColumns()
+ public void testColumns()
{
TableBuilder builder = TableBuilder.datasource("foo", "P1D");
// OK to have no columns
{
TableSpec spec = builder.copy()
.buildSpec();
- expectValidationSucceeds(spec);
+ ResolvedTable table = registry.resolve(spec);
+ table.validate();
+ DatasourceFacade facade = new DatasourceFacade(registry.resolve(table.spec()));
+ assertFalse(facade.hasRollup());
+ assertTrue(facade.columnFacades().isEmpty());
}
// OK to have no column type
{
TableSpec spec = builder.copy()
.column("foo", null)
.buildSpec();
- expectValidationSucceeds(spec);
+ ResolvedTable table = registry.resolve(spec);
+ table.validate();
+
+ DatasourceFacade facade = new DatasourceFacade(registry.resolve(table.spec()));
+ assertFalse(facade.hasRollup());
+ assertEquals(1, facade.columnFacades().size());
+ ColumnFacade col = facade.columnFacades().get(0);
+ assertSame(spec.columns().get(0), col.spec());
}
- // Time column can have no type
+ // Can have a legal scalar type
{
TableSpec spec = builder.copy()
- .column(Columns.TIME_COLUMN, null)
+ .column("foo", Columns.VARCHAR)
.buildSpec();
- expectValidationSucceeds(spec);
+ ResolvedTable table = registry.resolve(spec);
+ table.validate();
+ DatasourceFacade facade = new DatasourceFacade(registry.resolve(table.spec()));
+ assertEquals(1, facade.columnFacades().size());
+ ColumnFacade col = facade.columnFacades().get(0);
+ assertSame(spec.columns().get(0), col.spec());
}
- // Time column can only have TIMESTAMP type
+ // Reject an unknown SQL type
{
TableSpec spec = builder.copy()
- .timeColumn()
+ .column("foo", "BOGUS")
.buildSpec();
- expectValidationSucceeds(spec);
+ expectValidationFails(spec);
}
+
+ // Reject duplicate columns
{
TableSpec spec = builder.copy()
- .column(Columns.TIME_COLUMN, Columns.VARCHAR)
+ .column("foo", Columns.VARCHAR)
+ .column("bar", Columns.BIGINT)
.buildSpec();
- expectValidationFails(spec);
+ expectValidationSucceeds(spec);
}
-
- // Can have a legal scalar type
{
TableSpec spec = builder.copy()
.column("foo", Columns.VARCHAR)
+ .column("foo", Columns.BIGINT)
.buildSpec();
- expectValidationSucceeds(spec);
+ expectValidationFails(spec);
}
+ }
- // Reject an unknown SQL type
+ @Test
+ public void testRollup()
+ {
+ TableMetadata table = TableBuilder.datasource("foo", "P1D")
+ .column(Columns.TIME_COLUMN, "TIMESTAMP('PT1M')")
+ .column("a", null)
+ .column("b", Columns.VARCHAR)
+ .column("c", "SUM(BIGINT)")
+ .build();
+
+ table.validate();
+ List<ColumnSpec> columns = table.spec().columns();
+
+ assertEquals(4, columns.size());
+ assertEquals(Columns.TIME_COLUMN, columns.get(0).name());
+ assertEquals("TIMESTAMP('PT1M')", columns.get(0).sqlType());
+
+ assertEquals("a", columns.get(1).name());
+ assertNull(columns.get(1).sqlType());
+
+ assertEquals("b", columns.get(2).name());
+ assertEquals(Columns.VARCHAR, columns.get(2).sqlType());
+
+ assertEquals("c", columns.get(3).name());
+ assertEquals("SUM(BIGINT)", columns.get(3).sqlType());
+
+ DatasourceFacade facade = new DatasourceFacade(registry.resolve(table.spec()));
+ assertTrue(facade.hasRollup());
+ assertEquals(4, facade.columnFacades().size());
+ ColumnFacade col = facade.columnFacades().get(3);
Review Comment:
## Unread local variable
Variable 'ColumnFacade col' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/3598)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066563594
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefn.java:
##########
@@ -47,35 +46,30 @@ public class TableDefn extends ObjectDefn
*/
public static final String DESCRIPTION_PROPERTY = "description";
- private final Map<String, ColumnDefn> columnDefns;
+ private final Map<String, PropertyDefn<?>> columnProperties;
public TableDefn(
final String name,
final String typeValue,
final List<PropertyDefn<?>> properties,
- final List<ColumnDefn> columnDefns
+ final List<PropertyDefn<?>> columnProperties
)
{
super(
name,
typeValue,
CatalogUtils.concatLists(
Collections.singletonList(
- new ModelProperties.StringPropertyDefn(DESCRIPTION_PROPERTY, null)
+ new ModelProperties.StringPropertyDefn(DESCRIPTION_PROPERTY)
),
properties
)
);
- this.columnDefns = columnDefns == null ? Collections.emptyMap() : toColumnMap(columnDefns);
+ this.columnProperties = toPropertyMap(columnProperties);
}
- public static Map<String, ColumnDefn> toColumnMap(final List<ColumnDefn> colTypes)
+ public void bind(TableDefnRegistry registry)
Review Comment:
I assume this will be defined at some later time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067559789
##########
server/src/main/java/org/apache/druid/catalog/model/table/HttpInputSourceDefn.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.HttpInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.metadata.DefaultPasswordProvider;
+import org.apache.druid.metadata.EnvironmentVariablePasswordProvider;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Definition of an HTTP input source source.
+ * <p>
+ * Provides a parameterized form where the user defines a value for the
+ * {@code uriTemplate} table property in the table spec, then provides the partial URLs
+ * in a table function to use for that one query. The final URIs are created by combining
+ * the template and the arguments. Example:
+ * <li>{@code uriTemplate} property): "http://example.com/data/kttm-{}.json"</li>
+ * <li>{@code uris} function argument: "22-Nov-21, 22-Nov-22"</li>
+ * </ul>
+ * <p>
+ * When the template is used, the format is optional: it can be provided either with
+ * the table spec or at runtime, depending on what the user wants to accomplish. In the
+ * above, where the ".json" is encoded in the template, it makes sense to include the format
+ * with the spec. If the template was "http://example.com/data/{}", and the data comes in
+ * multiple formats, it might make sense to specify the format in the query. In this case,
+ * the table spec acts more like a connection.
+ * <p>
+ * If the template is not used, then the {@code uris} property must be provided in the
+ * table spec, along with the corresponding format.
+ * <p>
+ * The above semantics make a bit more sense when we realize that the spec can also
+ * provide a user name and password. When those are provided, then the input source must
+ * name a single site: the one for which the credentials are valid. Given this, the only
+ * table spec that makes sense is one where the URI is defined: either as a template or
+ * explicitly.
+ * <p>
+ * When used as an ad-hoc function, the user specifies the uris and optional user name
+ * and password: the template is not available (or useful) in the ad-hoc case.
+ * <p>
+ * Table function parameters are cleaned up relative to the input source field names to
+ * make them a bit easier to use.
+ */
+public class HttpInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = HttpInputSource.TYPE_KEY;
+
+ // Catalog properties that map to fields in the HttpInputSource. See
+ // that class for the meaning of these properties.
+
+ public static final String URI_TEMPLATE_PROPERTY = "uriTemplate";
+
+ public static final String URIS_PARAMETER = "uris";
+
+ // Note, cannot be the simpler "user" since USER is a reserved word in SQL
+ // and we don't want to require users to quote "user" each time it is used.
+ public static final String USER_PARAMETER = "userName";
+ public static final String PASSWORD_PARAMETER = "password";
+ public static final String PASSWORD_ENV_VAR_PARAMETER = "passwordEnvVar";
+
+ private static final List<ParameterDefn> URI_PARAMS = Arrays.asList(
+ new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true)
+ );
+
+ private static final List<ParameterDefn> USER_PWD_PARAMS = Arrays.asList(
+ new Parameter(USER_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(PASSWORD_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(PASSWORD_ENV_VAR_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the HttpInputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PASSWORD_FIELD = "httpAuthenticationPassword";
+ private static final String USERNAME_FIELD = "httpAuthenticationUsername";
+
+ @Override
+ public String typeValue()
+ {
+ return TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return HttpInputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUri = sourceMap.containsKey(URIS_FIELD);
+ final String uriTemplate = table.resolvedTable().stringProperty(URI_TEMPLATE_PROPERTY);
+ final boolean hasTemplate = uriTemplate != null;
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (!hasUri && !hasTemplate) {
+ throw new IAE(
+ "External HTTP tables must provide either a URI or a %s property",
+ URI_TEMPLATE_PROPERTY
+ );
+ }
+ if (hasUri && hasTemplate) {
+ throw new IAE(
+ "External HTTP tables must provide only one of a URI or a %s property",
+ URI_TEMPLATE_PROPERTY
+ );
+ }
+ if (hasUri && !hasFormat) {
+ throw new IAE(
+ "An external HTTP table with a URI must also provide the corresponding format"
+ );
+ }
+ if (hasUri && !hasColumns) {
+ throw new IAE(
+ "An external HTTP table with a URI must also provide the corresponding columns"
+ );
+ }
+ if (hasTemplate) {
+
+ // Verify the template
+ templateMatcher(uriTemplate);
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ try {
+ sourceMap.put(
+ URIS_FIELD,
+ Collections.singletonList(new URI("https://bogus.com/file"))
Review Comment:
Was this just added for debugging? Should remove?
##########
server/src/main/java/org/apache/druid/catalog/model/table/ExternalTableSpec.java:
##########
@@ -28,13 +28,13 @@
/**
* Catalog form of an external table specification used to pass along the three
* components needed for an external table in MSQ ingest. Just like
- * {@code ExternalTableSource}, except that the parameters are not required
- * to be non-null.
+ * {@code ExternalTableSource}, except that the class is visible in this
+ * module.
*/
public class ExternalTableSpec
{
- @Nullable public final InputSource inputSource;
- @Nullable public final InputFormat inputFormat;
+ public final InputSource inputSource;
Review Comment:
Are these fields guaranteed to be non-null now?
##########
server/src/main/java/org/apache/druid/catalog/model/table/ExternalTableDefn.java:
##########
@@ -19,254 +19,291 @@
package org.apache.druid.catalog.model.table;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import org.apache.druid.catalog.model.CatalogUtils;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.druid.catalog.model.ColumnSpec;
-import org.apache.druid.catalog.model.Columns;
-import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.ObjectPropertyDefn;
import org.apache.druid.catalog.model.ModelProperties.PropertyDefn;
-import org.apache.druid.catalog.model.ParameterizedDefn;
-import org.apache.druid.catalog.model.PropertyAttributes;
import org.apache.druid.catalog.model.ResolvedTable;
import org.apache.druid.catalog.model.TableDefn;
-import org.apache.druid.catalog.model.table.InputFormats.InputFormatDefn;
+import org.apache.druid.catalog.model.TableDefnRegistry;
import org.apache.druid.data.input.InputFormat;
import org.apache.druid.data.input.InputSource;
-import org.apache.druid.java.util.common.IAE;
-import org.apache.druid.java.util.common.ISE;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
+import java.util.Arrays;
import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
/**
- * Definition of an external input source, primarily for ingestion.
+ * Definition of an external table, primarily for ingestion.
* The components are derived from those for Druid ingestion: an
* input source, a format and a set of columns. Also provides
* properties, as do all table definitions.
+ *
+ * <h4>Partial Tables and Connections</h4>
+ *
+ * An input source is a template for an external table. The input
+ * source says how to get data, and optionally the format and structure of that data.
+ * Since Druid never ingests the same data twice, the actual external table needs
+ * details that says which data to read on any specific ingestion. Thus, an external
+ * table is usually a "partial table": all the information that remains constant
+ * across ingestions, but without the information that changes. The changing
+ * information is typically the list of files (or objects or URLs) to ingest.
+ * <p>
+ * The pattern is:<br>
+ * {@code external table spec + parameters --> external table}
+ * <p>
+ * Since an input source is a parameterized (partial) external table, we can reuse
+ * the table metadata structures and APIs, avoiding the need to have a separate (but
+ * otherwise identical) structure for external tables.
+ *
+ * An external table can be thought of as a "connection", though Druid does not
+ * use that term. When used as a connection, the external table spec will omit the
+ * format. Instead, the format will also be provided at ingest time, along with the
+ * list of tables (or objects.)
+ * <p>
+ * To keep all this straight, we adopt the following terms:
+ * <dl>
+ * <dt>External table spec</dt>
+ * <dd>The JSON serialized version of an external table which can be partial or
+ * complete. The spec is a named entry in the Druid catalog</dd>
+ * <dt>Complete spec</dt>
+ * <dd>An external table spec that provides all information needed to access an
+ * external table. Each use identifies the same set of data. Useful if MSQ is used
+ * to query an external data source. A complete spec can be referenced as a
+ * first-class table in a {@code FROM} clause in an MSQ query.</dd>
+ * <dt>Partial spec</dt>
+ * <dd>An external table spec that omits some information. That information must
+ * be provided at query time in the form of a {@code TABLE} function. If the partial
+ * spec includes a format, then it is essentially a <i>partial table</i>. If it
+ * omits the format, then it is essentially a <i>connection</i>.</dd>
+ * <dt>Completed table</dt>
+ * <dd>The full external table that results from a partial spec and a set of SQL
+ * table function parameters.</dd>
+ * <dt>Ad-hoc table</dt>
+ * <dd>Users can define an external table using the generic {@code EXTERN} function
+ * or one of the input-source-specific functions. In this case, there is no
+ * catalog entry: all information comes from the SQL table function</dd>
+ * <dt>Partial table function</dt>
+ * <dd>The SQL table function used to "complete" a partial spec. The function
+ * defines parameters to fill in the missing information. The function is generated
+ * on demand and has the same name as the catalog entry for the partial spec.
+ * The function will include parameters for format if the catalog spec does not
+ * specify a format. Else, the format parameters are omitted and the completed
+ * table uses the format provided in the catalog spec.</dd>
+ * <dt>Ad-hoc table function</dt>
+ * <dd>The SQL table function used to create an ad-hoc external table. The function
+ * as a name defined by the {@link InputFormatDefn}, and has parameters for all
+ * support formats: the user must specify all input source and format properties.</dd>
+ * </dl>
+ *
+ * <h4>External Table Structure</h4>
+ *
+ * The external table is generic: it represents all valid combinations of input
+ * sources and formats. Rather than have a different table definition for each, we
+ * instead split out input sources and formats into their own definitions, and those
+ * definitions are integrated and used by this external table class. As a result,
+ * the {@code properties} field will contain the {@code source} property which has
+ * the JSON serialized form of the input source (minus items to be parameterized.)
+ * <p>
+ * Similarly, if the external table also defines a format (rather than requiring the
+ * format at ingest time), then the {@code format} property holds the JSON-serialized
+ * form of the input format, minus columns. The columns can be provided in the spec,
+ * in the {@code columns} field. The {@link InputFormatDefn} converts the columns to
+ * the form needed by the input format.
+ * <p>
+ * Druid's input sources all require formats. However, some sources may not actually
+ * need the format. A JDBC input source for example, needs no format. In other cases,
+ * there may be a subset of formats. Each {@link InputSourceDefn} is responsible for
+ * working out which formats (if any) are required. This class is agnostic about whether
+ * the format is supplied. (Remember that, when used as a connection, the external table
+ * will provide no format until ingest time.)
+ * <p>
+ * By contrast, the input source is always required.
+ *
+ * <h4>Data Formats and Conversions</h4>
+ *
+ * Much of the code here handles conversion of an external table specification to
+ * the form needed by SQL. Since SQL is not visible here, we instead create an
+ * instance of {@link ExternalTableSpec} which holds the input source, input format
+ * and row signature in the form required by SQL.
+ * <p>
+ * This class handles table specifications in three forms:
+ * <ol>
+ * <li>From a fully-defined table specification, converted to a {@code ExternalTableSpec}
+ * by the {@link #convert(ResolvedTable)} function.</li>
+ * <li>From a fully-defined set of arguments to a SQL table function. The
+ * {@link InputSourceDefn#adHocTableFn()} method provides the function definition which
+ * handles the conversion.</li>
+ * <li>From a partially-defined table specification in the catalog, augmented by
+ * parameters passed from a SQL function. The {@link #tableFn(ResolvedTable)} method
+ * creates the required function by caching the table spec. That function then combines
+ * the parameters to produce the required {@code ExternalTableSpec}.</li>
+ * </ol>
+ * <p>
+ * To handle these formats, and the need to adjust JSON, conversion to an
+ * {@code ExternalTableSpec} occurs in multiple steps:
+ * <ul>
+ * <li>When using a table spec, the serialized JSON is first converted to a generic
+ * Java map: one for the input source, another for the format.</li>
+ * <li>When using a SQL function, the SQL arguments are converted (if needed) and
+ * written into a Java map. If the function references an existing table spec: then
+ * the JSON map is first populated with the deserialized spec.</li>
+ * <li>Validation and/or adjustments are made to the Java map. Adjustments are
+ * those described elsewhere in this Javadoc.</li>
+ * <li>The column specifications from either SQL or the table spec are converted to
+ * a list of column names, and placed into the Java map for the input format.</li>
+ * <li>The maps are converted to the {@link InputSource} or {@link InputFormat}
+ * objects using a Jackson conversion.</li>
+ * </ul>
+ * The actual conversions are handled in the {@link InputFormatDefn} and
+ * {@link InputFormatDefn} classes, either directly (for a fully-defined table
+ * function) or starting here (for other use cases).
+ *
+ * <h4>Property and Parameter Names</h4>
+ *
+ * Pay careful attention to names: the names may be different in each of the above
+ * cases:
+ * <ul>
+ * <li>The table specification stores the input source and input format specs using
+ * the names defined by the classes themselves. That is, the table spec holds a string
+ * that represents the Jackson-serialized form of those classes. In some cases, the
+ * JSON can be a subset: some sources and formats have obscure checks, or options which
+ * are not available via this path. The code that does conversions will adjust the JSON
+ * prior to conversion. Each JSON object has a type field: the value of that type
+ * field must match that defined in the Jackson annotations for the corresponding
+ * class.</li>
+ * <li>SQL table functions use argument names that are typically selected for user
+ * convenience, and may not be the same as the JSON field name. For example, a field
+ * name may be a SQL reserved word, or may be overly long, or may be obscure. The code
+ * for each input source and input format definition does the needed conversion.</li>
+ * <li>Each input source and input format has a type. The input format type is given,
+ * in SQL by the {@code format} property. The format type name is typically the same
+ * as the JSON type name, but need not be.</li>
+ * </ul>
+ *
+ * <h4>Extensions</h4>
+ *
+ * This class is designed to work both with "well known" Druid input sources and formats,
+ * and those defined in an extension. For extension-defined sources and formats to work,
+ * the extension must define an {@link InputSourceDefn} or {@link InputFormatDefn} which
+ * are put into the {@link TableDefnRegistry} and thus available to this class. The result
+ * is that this class is ignorant of the actual details of sources and formats: it instead
+ * delegates to the input source and input format definitions for that work.
* <p>
- * The external table implements the mechanism for parameterized tables,
- * but does not implement the {@link ParameterizedDefn} interface itself.
- * Tables which are parameterized implement that interface to expose
- * methods defined here.
+ * Input sources and input formats defined in an extension are considered "ephemeral":
+ * they can go away if the corresponding extension is removed from the system. In that
+ * case, any table functions defined by those extensions are no longer available, and
+ * any SQL statements that use those functions will no longer work. The catalog may contain
+ * an external table spec that references those definitions. Such specs will continue to
+ * reside in the catalog, and can be retrieved, but they will fail any query that attempts
+ * to reference them.
*/
-public abstract class ExternalTableDefn extends TableDefn
+public class ExternalTableDefn extends TableDefn
{
- public static final String EXTERNAL_COLUMN_TYPE = "extern";
-
- public abstract static class FormattedExternalTableDefn extends ExternalTableDefn
- {
- public static final String FORMAT_PROPERTY = "format";
-
- private final Map<String, InputFormatDefn> formats;
-
- public FormattedExternalTableDefn(
- final String name,
- final String typeValue,
- final List<PropertyDefn<?>> properties,
- final List<ColumnDefn> columnDefns,
- final List<InputFormatDefn> formats
- )
- {
- super(
- name,
- typeValue,
- addFormatProperties(properties, formats),
- columnDefns
- );
- ImmutableMap.Builder<String, InputFormatDefn> builder = ImmutableMap.builder();
- for (InputFormatDefn format : formats) {
- builder.put(format.typeTag(), format);
- }
- this.formats = builder.build();
- }
-
- /**
- * Add format properties to the base set, in the order of the formats,
- * in the order defined by the format. Allow same-named properties across
- * formats, as long as the types are the same.
- */
- private static List<PropertyDefn<?>> addFormatProperties(
- final List<PropertyDefn<?>> properties,
- final List<InputFormatDefn> formats
- )
- {
- List<PropertyDefn<?>> toAdd = new ArrayList<>();
- PropertyDefn<?> formatProp = new ModelProperties.StringPropertyDefn(FORMAT_PROPERTY, PropertyAttributes.SQL_FN_PARAM);
- toAdd.add(formatProp);
- Map<String, PropertyDefn<?>> formatProps = new HashMap<>();
- for (InputFormatDefn format : formats) {
- for (PropertyDefn<?> prop : format.properties()) {
- PropertyDefn<?> existing = formatProps.putIfAbsent(prop.name(), prop);
- if (existing == null) {
- toAdd.add(prop);
- } else if (existing.getClass() != prop.getClass()) {
- throw new ISE(
- "Format %s, property %s of class %s conflicts with another format property of class %s",
- format.name(),
- prop.name(),
- prop.getClass().getSimpleName(),
- existing.getClass().getSimpleName()
- );
- }
- }
- }
- return CatalogUtils.concatLists(properties, toAdd);
- }
-
- @Override
- protected InputFormat convertFormat(ResolvedTable table)
- {
- return formatDefn(table).convert(table);
- }
+ /**
+ * Identifier for external tables.
+ */
+ public static final String TABLE_TYPE = "extern";
- protected InputFormatDefn formatDefn(ResolvedTable table)
- {
- String formatTag = table.stringProperty(FORMAT_PROPERTY);
- if (formatTag == null) {
- throw new IAE("%s property must be set", FORMAT_PROPERTY);
- }
- InputFormatDefn formatDefn = formats.get(formatTag);
- if (formatDefn == null) {
- throw new IAE(
- "Format type [%s] for property %s is not valid",
- formatTag,
- FORMAT_PROPERTY
- );
- }
- return formatDefn;
- }
+ /**
+ * Column type for external tables.
+ */
+ public static final String EXTERNAL_COLUMN_TYPE = "extern";
Review Comment:
What is this used for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066589250
##########
server/src/main/java/org/apache/druid/catalog/model/table/DatasourceDefn.java:
##########
@@ -19,65 +19,137 @@
package org.apache.druid.catalog.model.table;
+import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
import org.apache.druid.catalog.model.ColumnSpec;
import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.GranularityPropertyDefn;
+import org.apache.druid.catalog.model.ModelProperties.StringListPropertyDefn;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefn;
import org.apache.druid.java.util.common.IAE;
import org.apache.druid.java.util.common.StringUtils;
-import java.util.Collections;
+import java.util.Arrays;
+import java.util.List;
-/**
- * Definition of a Druid datasource. The datasource may use rollup, but rollup
- * is not represented in the catalog: it is just a way that applications store
- * data into a table.
- */
-public class DatasourceDefn extends AbstractDatasourceDefn
+public class DatasourceDefn extends TableDefn
{
/**
- * Definition of a column in a datasource.
+ * Segment grain at ingestion and initial compaction. Aging rules
+ * may override the value as segments age. If not provided here,
+ * then it must be provided at ingestion time.
*/
- public static class DatasourceColumnDefn extends ColumnDefn
- {
- public static final String COLUMN_TYPE = "column";
+ public static final String SEGMENT_GRANULARITY_PROPERTY = "segmentGranularity";
+
+ /**
+ * Catalog property value for the "all time" granularity.
+ */
+ public static final String ALL_GRANULARITY = "ALL";
+
+ /**
+ * The target segment size at ingestion and initial compaction.
+ * If unset, then the system setting is used.
+ */
+ public static final String TARGET_SEGMENT_ROWS_PROPERTY = "targetSegmentRows";
+
+ /**
+ * The clustering column names and sort order for each new segment.
+ */
+ public static final String CLUSTER_KEYS_PROPERTY = "clusterKeys";
+
+ /**
+ * The set of existing columns to "delete" (actually, just hide) from the
+ * SQL layer. Used to "remove" unwanted columns to avoid the need to rewrite
+ * existing segments to accomplish the task.
+ */
+ public static final String HIDDEN_COLUMNS_PROPERTY = "hiddenColumns";
+
+ public static final String SEALED_PROPERTY = "sealed";
Review Comment:
docs for this property would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1071599126
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidCalciteSchemaModule.java:
##########
@@ -64,7 +64,7 @@ public void configure(Binder binder)
}
@Provides
- @Singleton
+ @LazySingleton
Review Comment:
@imply-cheddar and I had an offline conversation to track down the `@LazySingleton` issues and he kindly provided a PR to resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066585225
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ return new ExternalTableSpec(
+ convertArgsToSource(args, jsonMapper),
+ convertArgsToFormat(args, columns, jsonMapper),
+ Columns.convertSignature(columns)
+ );
+ }
+
+ /**
+ * Convert the input source using arguments to a "from scratch" table function.
+ */
+ protected InputSource convertArgsToSource(Map<String, Object> args, ObjectMapper jsonMapper)
+ {
+ final Map<String, Object> jsonMap = new HashMap<>();
+ auditInputSource(jsonMap);
+ convertArgsToSourceMap(jsonMap, args);
+ return convertSource(jsonMap, jsonMapper);
+ }
+
+ /**
+ * Convert SQL arguments to the corresponding "generic JSON" form in the given map.
+ * The map will then be adjusted and converted to the actual input source.
+ */
+ protected abstract void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args);
+
+ /**
+ * Convert SQL arguments, and the column schema, to an input format, if required.
+ */
+ protected InputFormat convertArgsToFormat(Map<String, Object> args, List<ColumnSpec> columns, ObjectMapper jsonMapper)
+ {
+ return null;
+ }
+
+ /**
+ * Complete a partial table using the table function arguments and columns provided.
+ * The arguments match the set of parameters used for the function. The columns are
+ * provided if the SQL included an {@code EXTENDS} clause: the implementation should decide
+ * if columns are required (or allowed) depending on whether the partial spec already
+ * defines columns.
+ *
+ * @param table the partial table spec, with input source and format parsed into a
+ * generic Java map
+ * @param args the argument values provided in the SQL table function call. The arguments
+ * use the Java types defined in the parameter definitions.
+ * @param columns the set of columns (if any) from the SQL {@code EXTEND} clause
+ *
+ * @return an external table spec which Calcite can consume
+ */
+ protected abstract ExternalTableSpec convertCompletedTable(
Review Comment:
This this be moved into the `PartialTableFunction` class, since it seems it should only be used in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067199081
##########
server/src/main/java/org/apache/druid/catalog/model/table/ExternalTableDefn.java:
##########
@@ -19,254 +19,289 @@
package org.apache.druid.catalog.model.table;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import org.apache.druid.catalog.model.CatalogUtils;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.druid.catalog.model.ColumnSpec;
-import org.apache.druid.catalog.model.Columns;
-import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.ObjectPropertyDefn;
import org.apache.druid.catalog.model.ModelProperties.PropertyDefn;
-import org.apache.druid.catalog.model.ParameterizedDefn;
-import org.apache.druid.catalog.model.PropertyAttributes;
import org.apache.druid.catalog.model.ResolvedTable;
import org.apache.druid.catalog.model.TableDefn;
-import org.apache.druid.catalog.model.table.InputFormats.InputFormatDefn;
+import org.apache.druid.catalog.model.TableDefnRegistry;
import org.apache.druid.data.input.InputFormat;
import org.apache.druid.data.input.InputSource;
-import org.apache.druid.java.util.common.IAE;
-import org.apache.druid.java.util.common.ISE;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
+import java.util.Arrays;
import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
/**
- * Definition of an external input source, primarily for ingestion.
+ * Definition of an external table, primarily for ingestion.
* The components are derived from those for Druid ingestion: an
* input source, a format and a set of columns. Also provides
* properties, as do all table definitions.
+ *
+ * <h4>Partial Tables and Connections</h4>
+ *
+ * An input source is a template for an external table. The input
+ * source says how to get data, and optionally the format and structure of that data.
+ * Since Druid never ingests the same data twice, the actual external table needs
+ * details that says which data to read on any specific ingestion. Thus, an external
+ * table is usually a "partial table": all the information that remains constant
+ * across ingestions, but without the information that changes. The changing
+ * information is typically the list of files (or objects or URLs) to ingest.
+ * <p>
+ * The pattern is:<br>
+ * {@code external table spec + parameters --> external table}
+ * <p>
+ * Since an input source is a parameterized (partial) external table, we can reuse
+ * the table metadata structures and APIs, avoiding the need to have a separate (but
+ * otherwise identical) structure for external tables.
+ *
+ * An external table can be thought of as a "connection", though Druid does not
+ * use that term. When used as a connection, the external table spec will omit the
+ * format. Instead, the format will also be provided at ingest time, along with the
+ * list of tables (or objects.)
+ * <p>
+ * To keep all this straight, we adopt the following terms:
+ * <dl>
+ * <dt>External table spec</dt>
+ * <dd>The JSON serialized version of an external table which can be partial or
+ * complete. The spec is a named entry in the Druid catalog</dd>
+ * <dt>Complete spec</dt>
+ * <dd>An external table spec that provides all information needed to access an
+ * external table. Each use identifies the same set of data. Useful if MSQ is used
+ * to query an external data source. A complete spec can be referenced as a
+ * first-class table in a {@code FROM} clause in an MSQ query.</dd>
+ * <dt>Partial spec</dt>
+ * <dd>An external table spec that omits some information. That information must
+ * be provided at query time in the form of a {@code TABLE} function. If the partial
+ * spec includes a format, then it is essentially a <i>partial table</i>. If it
+ * omits the format, then it is essentially a <i>connection</i>.</dd>
+ * <dt>Completed table</dt>
+ * <dd>The full external table that results from a partial spec and a set of SQL
+ * table function parameters.</dd>
+ * <dt>Ad-hoc table</dt>
+ * <dd>Users can define an external table using the generic {@code EXTERN} function
+ * or one of the input-source-specific functions. In this case, there is no
+ * catalog entry: all information comes from the SQL table function</dd>
+ * <dt>Partial table function</dt>
+ * <dd>The SQL table function used to "complete" a partial spec. The function
+ * defines parameters to fill in the missing information. The function is generated
+ * on demand and has the same name as the catalog entry for the partial spec.
+ * The function will include parameters for format if the catalog spec does not
+ * specify a format. Else, the format parameters are omitted and the completed
+ * table uses the format provided in the catalog spec.</dd>
+ * <dt>Ad-hoc table function</dt>
+ * <dd>The SQL table function used to create an ad-hoc external table. The function
+ * as a name defined by the {@link InputFormatDefn}, and has parameters for all
+ * support formats: the user must specify all input source and format properties.</dd>
+ * </dl>
+ *
+ * <h4>External Table Structure</h4>
+ *
+ * The external table is generic: it represents all valid combinations of input
+ * sources and formats. Rather than have a different table definition for each, we
+ * instead split out input sources and formats into their own definitions, and those
+ * definitions are integrated and used by this external table class. As a result,
+ * the {@code properties} field will contain the {@code source} property which has
+ * the JSON serialized form of the input source (minus items to be parameterized.)
+ * <p>
+ * Similarly, if the external table also defines a format (rather than requiring the
+ * format at ingest time), then the {@code format} property holds the JSON-serialized
+ * form of the input format, minus columns. The columns can be provided in the spec,
+ * in the {@code columns} field. The {@link InputFormatDefn} converts the columns to
+ * the form needed by the input format.
+ * <p>
+ * Druid's input sources all require formats. However, some sources may not actually
+ * need the format. A JDBC input source for example, needs no format. In other cases,
+ * there may be a subset of formats. Each {@link InputSourceDefn} is responsible for
+ * working out which formats (if any) are required. This class is agnostic about whether
+ * the format is supplied. (Remember that, when used as a connection, the external table
+ * will provide no format until ingest time.)
+ * <p>
+ * By contrast, the input source is always required.
+ *
+ * <h4>Data Formats and Conversions</h4>
+ *
+ * Much of the code here handles conversion of an external table specification to
+ * the form needed by SQL. Since SQL is not visible here, we instead create an
+ * instance of {@link ExternalTableSpec} which holds the input source, input format
+ * and row signature in the form required by SQL.
+ * <p>
+ * This class handles table specifications in three forms:
+ * <ol>
+ * <li>From a fully-defined table specification, converted to a {@code ExternalTableSpec}
+ * by the {@link #convert(ResolvedTable)} function.</li>
+ * <li>From a fully-defined set of arguments to a SQL table function. The
+ * {@link InputSourceDefn#adHocTableFn()} method provides the function definition which
+ * handles the conversion.</li>
+ * <li>From a partially-defined table specification in the catalog, augmented by
+ * parameters passed from a SQL function. The {@link #tableFn(ResolvedTable)} method
+ * creates the required function by caching the table spec. That function then combines
+ * the parameters to produce the required {@code ExternalTableSpec}.</li>
+ * </ol>
+ * <p>
+ * To handle these formats, and the need to adjust JSON, conversion to an
+ * {@code ExternalTableSpec} occurs in multiple steps:
+ * <ul>
+ * <li>When using a table spec, the serialized JSON is first converted to a generic
+ * Java map: one for the input source, another for the format.</li>
+ * <li>When using a SQL function, the SQL arguments are converted (if needed) and
+ * written into a Java map. If the function references an existing table spec: then
+ * the JSON map is first populated with the deserialized spec.</li>
+ * <li>Validation and/or adjustments are made to the Java map. Adjustments are
+ * those described elsewhere in this Javadoc.</li>
+ * <li>The column specifications from either SQL or the table spec are converted to
+ * a list of column names, and placed into the Java map for the input format.</li>
+ * <li>The maps are converted to the {@link InputSource} or {@link InputFormat}
+ * objects using a Jackson conversion.</li>
+ * </ul>
+ * The actual conversions are handled in the {@link InputFormatDefn} and
+ * {@link InputFormatDefn} classes, either directly (for a fully-defined table
+ * function) or starting here (for other use cases).
+ *
+ * <h4>Property and Parameter Names</h4>
+ *
+ * Pay careful attention to names: the names may be different in each of the above
+ * cases:
+ * <ul>
+ * <li>The table specification stores the input source and input format specs using
+ * the names defined by the classes themselves. That is, the table spec holds a string
+ * that represents the Jackson-serialized form of those classes. In some cases, the
+ * JSON can be a subset: some sources and formats have obscure checks, or options which
+ * are not available via this path. The code that does conversions will adjust the JSON
+ * prior to conversion. Each JSON object has a type field: the value of that type
+ * field must match that defined in the Jackson annotations for the corresponding
+ * class.</li>
+ * <li>SQL table functions use argument names that are typically selected for user
+ * convenience, and may not be the same as the JSON field name. For example, a field
+ * name may be a SQL reserved word, or may be overly long, or may be obscure. The code
+ * for each input source and input format definition does the needed conversion.</li>
+ * <li>Each input source and input format has a type. The input format type is given,
+ * in SQL by the {@code format} property. The format type name is typically the same
+ * as the JSON type name, but need not be.</li>
+ * </ul>
+ *
+ * <h4>Extensions</h4>
+ *
+ * This class is designed to work both with "well known" Druid input sources and formats,
+ * and those defined in an extension. For extension-defined sources and formats to work,
+ * the extension must define an {@link InputSourceDefn} or {@link InputFormatDefn} which
+ * are put into the {@link TableDefnRegistry} and thus available to this class. The result
+ * is that this class is ignorant of the actual details of sources and formats: it instead
+ * delegates to the input source and input format definitions for that work.
* <p>
- * The external table implements the mechanism for parameterized tables,
- * but does not implement the {@link ParameterizedDefn} interface itself.
- * Tables which are parameterized implement that interface to expose
- * methods defined here.
+ * Input sources and input formats defined in an extension are considered "ephemeral":
+ * they can go away if the corresponding extension is removed from the system. In that
+ * case, any table functions defined by those extensions are no longer available, and
+ * any SQL statements that use those functions will no longer work. The catalog may contain
+ * an external table spec that references those definitions. Such specs will continue to
+ * reside in the catalog, and can be retrieved, but they will fail any query that attempts
+ * to reference them.
*/
-public abstract class ExternalTableDefn extends TableDefn
+public class ExternalTableDefn extends TableDefn
{
- public static final String EXTERNAL_COLUMN_TYPE = "extern";
-
- public abstract static class FormattedExternalTableDefn extends ExternalTableDefn
- {
- public static final String FORMAT_PROPERTY = "format";
-
- private final Map<String, InputFormatDefn> formats;
-
- public FormattedExternalTableDefn(
- final String name,
- final String typeValue,
- final List<PropertyDefn<?>> properties,
- final List<ColumnDefn> columnDefns,
- final List<InputFormatDefn> formats
- )
- {
- super(
- name,
- typeValue,
- addFormatProperties(properties, formats),
- columnDefns
- );
- ImmutableMap.Builder<String, InputFormatDefn> builder = ImmutableMap.builder();
- for (InputFormatDefn format : formats) {
- builder.put(format.typeTag(), format);
- }
- this.formats = builder.build();
- }
-
- /**
- * Add format properties to the base set, in the order of the formats,
- * in the order defined by the format. Allow same-named properties across
- * formats, as long as the types are the same.
- */
- private static List<PropertyDefn<?>> addFormatProperties(
- final List<PropertyDefn<?>> properties,
- final List<InputFormatDefn> formats
- )
- {
- List<PropertyDefn<?>> toAdd = new ArrayList<>();
- PropertyDefn<?> formatProp = new ModelProperties.StringPropertyDefn(FORMAT_PROPERTY, PropertyAttributes.SQL_FN_PARAM);
- toAdd.add(formatProp);
- Map<String, PropertyDefn<?>> formatProps = new HashMap<>();
- for (InputFormatDefn format : formats) {
- for (PropertyDefn<?> prop : format.properties()) {
- PropertyDefn<?> existing = formatProps.putIfAbsent(prop.name(), prop);
- if (existing == null) {
- toAdd.add(prop);
- } else if (existing.getClass() != prop.getClass()) {
- throw new ISE(
- "Format %s, property %s of class %s conflicts with another format property of class %s",
- format.name(),
- prop.name(),
- prop.getClass().getSimpleName(),
- existing.getClass().getSimpleName()
- );
- }
- }
- }
- return CatalogUtils.concatLists(properties, toAdd);
- }
-
- @Override
- protected InputFormat convertFormat(ResolvedTable table)
- {
- return formatDefn(table).convert(table);
- }
+ /**
+ * Identifier for external tables.
+ */
+ public static final String TABLE_TYPE = "extern";
- protected InputFormatDefn formatDefn(ResolvedTable table)
- {
- String formatTag = table.stringProperty(FORMAT_PROPERTY);
- if (formatTag == null) {
- throw new IAE("%s property must be set", FORMAT_PROPERTY);
- }
- InputFormatDefn formatDefn = formats.get(formatTag);
- if (formatDefn == null) {
- throw new IAE(
- "Format type [%s] for property %s is not valid",
- formatTag,
- FORMAT_PROPERTY
- );
- }
- return formatDefn;
- }
+ /**
+ * Column type for external tables.
+ */
+ public static final String EXTERNAL_COLUMN_TYPE = "extern";
- @Override
- public void validate(ResolvedTable table)
- {
- super.validate(table);
- formatDefn(table).validate(table);
- List<ColumnSpec> columns = table.spec().columns();
- if (columns == null || columns.isEmpty()) {
- throw new IAE(
- "An external table of type %s must specify one or more columns",
- table.spec().type()
- );
- }
- }
- }
+ /**
+ * Property which holds the input source specification as serialized as JSON.
+ */
+ public static final String SOURCE_PROPERTY = "source";
/**
- * Definition of a column in a detail (non-rollup) datasource.
+ * Property which holds the optional input format specification, serialized as JSON.
*/
- public static class ExternalColumnDefn extends ColumnDefn
- {
- public ExternalColumnDefn()
- {
- super(
- "Column",
- EXTERNAL_COLUMN_TYPE,
- null
- );
- }
+ public static final String FORMAT_PROPERTY = "format";
- @Override
- public void validate(ColumnSpec spec, ObjectMapper jsonMapper)
- {
- super.validate(spec, jsonMapper);
- validateScalarColumn(spec);
- }
- }
+ /**
+ * Definition of the input source property.
+ */
+ private static final PropertyDefn<InputSource> SOURCE_PROPERTY_DEFN =
+ new ObjectPropertyDefn<InputSource>(SOURCE_PROPERTY, InputSource.class);
- protected static final ExternalColumnDefn INPUT_COLUMN_DEFN = new ExternalColumnDefn();
+ /**
+ * Definition of the input format property.
+ */
+ private static final PropertyDefn<InputFormat> FORMAT_PROPERTY_DEFN =
+ new ObjectPropertyDefn<InputFormat>(FORMAT_PROPERTY, InputFormat.class);
- private final List<PropertyDefn<?>> fields;
+ /**
+ * Type reference used to deserialize JSON to a generic map.
+ */
+ @VisibleForTesting
+ public static final TypeReference<Map<String, Object>> MAP_TYPE_REF = new TypeReference<Map<String, Object>>() { };
- public ExternalTableDefn(
- final String name,
- final String typeValue,
- final List<PropertyDefn<?>> fields,
- final List<ColumnDefn> columnDefns
- )
- {
- super(name, typeValue, fields, columnDefns);
- this.fields = fields;
- }
+ private TableDefnRegistry registry;
- public List<PropertyDefn<?>> parameters()
+ public ExternalTableDefn()
{
- return fields.stream()
- .filter(f -> PropertyAttributes.isExternTableParameter(f))
- .collect(Collectors.toList());
+ super(
+ "External table",
+ TABLE_TYPE,
+ Arrays.asList(
+ SOURCE_PROPERTY_DEFN,
+ FORMAT_PROPERTY_DEFN
+ ),
+ null
+ );
}
- public List<PropertyDefn<?>> tableFunctionParameters()
+ @Override
+ public void bind(TableDefnRegistry registry)
{
- return fields.stream()
- .filter(f -> PropertyAttributes.isSqlFunctionParameter(f))
- .collect(Collectors.toList());
+ this.registry = registry;
}
- /**
- * Merge parameters provided by a SQL table function with the catalog information
- * provided in the resolved table to produce a new resolved table used for a
- * specific query.
- */
- public abstract ResolvedTable mergeParameters(ResolvedTable table, Map<String, Object> values);
-
- public ExternalTableSpec convertToExtern(ResolvedTable table)
+ @Override
+ public void validate(ResolvedTable table)
{
- return new ExternalTableSpec(
- convertSource(table),
- convertFormat(table),
- Columns.convertSignature(table.spec())
- );
+ for (PropertyDefn<?> propDefn : properties().values()) {
+ if (propDefn != SOURCE_PROPERTY_DEFN && propDefn != FORMAT_PROPERTY_DEFN) {
Review Comment:
should we use `.equals` here instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067363578
##########
docs/multi-stage-query/reference.md:
##########
@@ -88,93 +118,260 @@ FROM TABLE(
http(
userName => 'bob',
password => 'secret',
- uris => 'http:foo.com/bar.csv',
+ uris => ARRAY['http:example.com/foo.csv', 'http:example.com/bar.csv'],
format => 'csv'
)
) (x VARCHAR, y VARCHAR, z BIGINT)
```
+#### Function Arguments
-The set of table functions and formats is preliminary in this release.
+These table functions are intended for use with the SQL by-name argument syntax
+as shown above. Because the functions include all parameters for all formats,
+using positional calls is both cumbersome and error-prone.
+
+Function argument names are generally the same as the JSON field names, except
+as noted below. Each argument has a SQL type which matches the JSON type. For
+arguments that take a string list in JSON, use the SQL `ARRAY[...]` syntax in
+SQL as shown in the above example.
+
+Array parameters are good candidates for use in parameterized queries. That is:
+
+```sql
+SELECT
+ <column>
+FROM TABLE(
+ http(
+ userName => 'bob',
+ password => 'secret',
+ uris => ?,
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR, z BIGINT)
+```
-#### `HTTP`
+Provide the list of URIs (in this case) as a query parameter in each ingest. Doing
+so is simpler than writing a script to insert the array into the SQL text.
-The `HTTP` table function represents the `HttpInputSource` class in Druid which allows you to
-read from an HTTP server. The function accepts the following arguments:
+#### `HTTP` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `userName` | Basic authentication user name | `httpAuthenticationUsername` | No |
-| `password` | Basic authentication password | `httpAuthenticationPassword` | No |
-| `passwordEnvVar` | Environment variable that contains the basic authentication password| `httpAuthenticationPassword` | No |
-| `uris` | Comma-separated list of URIs to read. | `uris` | Yes |
+The `HTTP` table function represents the
+[HTTP input source](../ingestion/native-batch-input-sources.md#http-input-source)
+to read from an HTTP server. The function accepts the following arguments:
-#### `INLINE`
+* `userName` (`VARCHAR`) - Same as JSON `httpAuthenticationUsername`.
+* `password` (`VARCHAR`) - Same as`httpAuthenticationPassword` when used with the default option.
+* `passwordEnvVar` (`VARCHAR`) - Same as the HTTP `httpAuthenticationPassword` when used with
+ the `"type": "environment"` option.
+* `uris` (`ARRAY` of `VARCHAR`)
-The `INLINE` table function represents the `InlineInputSource` class in Druid which provides
-data directly in the table function. The function accepts the following arguments:
+#### `INLINE` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `data` | Text lines of inline data. Separate lines with a newline. | `data` | Yes |
+The `INLINE` table function represents the
+[Inline input source](../ingestion/native-batch-input-sources.md#inline-input-source)
+which provides data directly in the table function. Parameter:
-#### `LOCALFILES`
+* `data` (`ARRAY` of `VARCHAR`) - Data lines, without a trailing newline, as an array.
-The `LOCALFILES` table function represents the `LocalInputSource` class in Druid which reads
+Example:
+
+```sql
+FROM TABLE(
+ inline(
+ data => ARRAY[
+ "a,b",
+ "c,d"],
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR)
+```
+
+
+#### `LOCALFILES` Function
+
+The `LOCALFILES` table function represents the
+[Local input source](../ingestion/native-batch-input-sources.md#local-input-source) which reads
files from the file system of the node running Druid. This is most useful for single-node
-installations. The function accepts the following arguments:
+installations. The function accepts the following parameters:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `baseDir` | Directory to read from. | `baseDir` | No |
-| `filter` | Filter pattern to read. Example: `*.csv`. | `filter` | No |
-| `files` | Comma-separated list of files to read. | `files` | No |
+* `baseDir`
+* `filter`
+* `files`
-You must either provide the `baseDir` or the list of `files`. You can provide both, in which case
-the files are assumed relative to the `baseDir`. If you provide a `filter`, you must provide the
-`baseDir`.
+When the local files input source is used directly in an `extern` function, or ingestion spec, you
+can provide either `baseDir` and `filter` or `files` but not both. This function, however, allows
+you to provide any of the following combinations:
-#### Table Function Format
+* `baseDir` - Matches all files in the given directory. (Assumes the filter is `*`.)
+* `baseDir` and `filter` - Match files in the given directory using the filter.
+* `baseDir` and `files` - A set of files relative to `baseDir`.
+* `files` - The files should be absolute paths, else they will be computed relative to Druid's
+ working directory (usually the Druid install directory.)
-Each of the table functions above requires that you specify a format.
+Examples:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `format` | The input format, using the same names as for `EXTERN`. | `inputFormat.type` | Yes |
+```sql
+ -- All files in /tmp, which must be CSV files
+ localfiles(baseDir => '/tmp',
+ format => 'csv')
+
+ -- CSV files in /tmp
+ localfiles(baseDir => '/tmp',
+ filter => '*.csv',
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(baseDir => '/tmp',
+ files => ARRAY['a.csv', 'b.csv'],
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(files => ARRAY['/tmp/a.csv', '/tmp/b.csv'],
+ format => 'csv')
+```
-#### CSV Format
+#### `S3` Function
-Use the `csv` format to read from CSV. This choice selects the Druid `CsvInputFormat` class.
+The `S3` table function represents the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) which reads
+files from an S3 bucket. The function accepts the following parameters to specify the
+objects to read:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+* `uris` (`ARRAY` of `VARCHAR`)
+* `prefix` (`VARCHAR`) - Corresponds to the JSON `prefixes` property, but allows a single
+ prefix.
+* `bucket` (`VARCHAR`) - Corresponds to the `bucket` field of the `objects` JSON field. SQL
+ does not have syntax for an array of objects. Instead, this function taks a single bucket,
+ and one or more objects within that bucket.
+* `paths` (`ARRAY` of `VARCHAR`) - Corresponds to the `path` fields of the `object` JSON field.
Review Comment:
They already appear? Check down two paragraphs. The intro to this paragraph says, "The function accepts the following parameters to specify the objects to read" while the later one says "also accepts the following security properties".
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
Review Comment:
Because this is a property (a top-level entry in the properties map for an external table definition), and not a field (a member of an input source object). Added a description of the naming convention to `InputSourceDefn`.
##########
docs/multi-stage-query/reference.md:
##########
@@ -280,6 +481,31 @@ The following ISO 8601 periods are supported for `TIME_FLOOR`:
- P3M
- P1Y
+The string constant can also include any of the keywords mentioned above:
+
+- `HOUR` - Same as `'PT1H'`
+- `DAY` - Same as `'P1D'`
Review Comment:
Removed this change because it will come in a later PR. Any of Druid's valid strings will work. However, WEEK isn't a valid SQL keyword, so it can't be used in the syntax shown earlier.
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECTS_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
+ private static final List<ParameterDefn> SECURITY_PARAMS = Arrays.asList(
+ new Parameter(ACCESS_KEY_ID_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(SECRET_ACCESS_KEY_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(ASSUME_ROLE_ARN_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the S3InputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PREFIXES_FIELD = "prefixes";
+ private static final String OBJECTS_FIELD = "objects";
+ private static final String OBJECT_GLOB_FIELD = "objectGlob";
+ private static final String PROPERTIES_FIELD = "properties";
+ private static final String ACCESS_KEY_ID_FIELD = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY_FIELD = "secretAccessKey";
+ private static final String ASSUME_ROLE_ARN_FIELD = "assumeRoleArn";
+
+ @Override
+ public String typeValue()
+ {
+ return S3StorageDruidModule.SCHEME;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return S3InputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (hasFormat && !hasColumns) {
+ throw new IAE(
+ "An external S3 table with a format must also provide the corresponding columns"
+ );
+ }
+
+ // The user can either provide a bucket, or can provide one of the valid items.
+ final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
+ final boolean hasBucket = bucket != null;
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
+ final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
+ final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
+ final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
+ if (hasBucket) {
+ if (hasUris || hasPrefix || hasObjects) {
+ throw new IAE(
+ "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
+ BUCKET_PROPERTY,
+ URIS_FIELD,
+ PREFIXES_FIELD,
+ OBJECTS_FIELD
+ );
+ }
+ if (hasGlob) {
+ throw new IAE(
+ "The %s property cannot be provided when the the %s property is set",
+ OBJECT_GLOB_FIELD,
+ BUCKET_PROPERTY
+ );
+ }
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ sourceMap.put(URIS_FIELD, Collections.singletonList(bucket));
+ }
+ super.validate(table);
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return CatalogUtils.concatLists(
+ Arrays.asList(
+ URI_PARAM_DEFN,
+ PREFIXES_PARAM_DEFN,
+ BUCKET_PARAM_DEFN,
+ PATHS_PARAM_DEFN,
+ OBJECTS_GLOB_PARAM_DEFN
+ ),
+ SECURITY_PARAMS
+ );
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, S3StorageDruidModule.SCHEME);
+ final List<String> uris = CatalogUtils.getStringArray(args, URIS_PARAMETER);
+ final List<String> prefixes = CatalogUtils.getStringArray(args, PREFIXES_PARAMETER);
+ final String bucket = CatalogUtils.getNonBlankString(args, BUCKET_PARAMETER);
+ final List<String> paths = CatalogUtils.getStringArray(args, PATHS_PARAMETER);
+ final String objectGlob = CatalogUtils.getNonBlankString(args, OBJECT_GLOB_PARAMETER);
+ final boolean hasUris = uris != null;
+ final boolean hasPrefixes = prefixes != null;
+ final boolean hasBucket = bucket != null;
+ final boolean hasPaths = !CollectionUtils.isNullOrEmpty(paths);
+ if (hasPaths && !hasBucket) {
+ throw new IAE(
+ "S3 requires the %s parameter if %s is set",
+ BUCKET_PARAMETER,
+ PATHS_PARAMETER
+ );
+ }
+ if ((hasUris && (hasPrefixes || hasBucket)) || (hasPrefixes && hasBucket)) {
+ throw new IAE(
+ "S3 accepts only one of %s, %s or %s",
+ PATHS_PARAMETER,
+ BUCKET_PARAMETER,
+ PREFIXES_PARAMETER
+ );
+ }
+ if (!hasUris && !hasPrefixes && !hasBucket) {
+ throw new IAE(
+ "S3 requires one of %s, %s or %s",
+ PATHS_PARAMETER,
+ BUCKET_PARAMETER,
+ PREFIXES_PARAMETER
+ );
+ }
+ if (hasUris) {
+ jsonMap.put(URIS_FIELD, CatalogUtils.stringListToUriList(uris));
+ }
+ if (hasPrefixes) {
+ jsonMap.put(PREFIXES_FIELD, prefixes);
+ }
+ if (hasBucket) {
+ if (!hasPaths) {
+ throw new IAE("When using the %s parameter, %s must also be provided", BUCKET_PARAMETER, PATHS_PARAMETER);
+ }
+ List<CloudObjectLocation> objects = new ArrayList<>();
+ for (String obj : paths) {
+ objects.add(new CloudObjectLocation(bucket, obj));
+ }
+ jsonMap.put(OBJECTS_FIELD, objects);
+ }
+ if (objectGlob != null) {
+ jsonMap.put(OBJECT_GLOB_FIELD, objectGlob);
Review Comment:
I don't think so. It's been a while since I did this mapping; I'd have to go back and plow through the code again to reverify what the S3 input source does and does not allow. What would it mean to provide a list of files and a filter?
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ return new ExternalTableSpec(
+ convertArgsToSource(args, jsonMapper),
+ convertArgsToFormat(args, columns, jsonMapper),
+ Columns.convertSignature(columns)
+ );
+ }
+
+ /**
+ * Convert the input source using arguments to a "from scratch" table function.
+ */
+ protected InputSource convertArgsToSource(Map<String, Object> args, ObjectMapper jsonMapper)
+ {
+ final Map<String, Object> jsonMap = new HashMap<>();
+ auditInputSource(jsonMap);
+ convertArgsToSourceMap(jsonMap, args);
+ return convertSource(jsonMap, jsonMapper);
+ }
+
+ /**
+ * Convert SQL arguments to the corresponding "generic JSON" form in the given map.
+ * The map will then be adjusted and converted to the actual input source.
+ */
+ protected abstract void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args);
+
+ /**
+ * Convert SQL arguments, and the column schema, to an input format, if required.
+ */
+ protected InputFormat convertArgsToFormat(Map<String, Object> args, List<ColumnSpec> columns, ObjectMapper jsonMapper)
+ {
+ return null;
+ }
+
+ /**
+ * Complete a partial table using the table function arguments and columns provided.
+ * The arguments match the set of parameters used for the function. The columns are
+ * provided if the SQL included an {@code EXTENDS} clause: the implementation should decide
+ * if columns are required (or allowed) depending on whether the partial spec already
+ * defines columns.
+ *
+ * @param table the partial table spec, with input source and format parsed into a
+ * generic Java map
+ * @param args the argument values provided in the SQL table function call. The arguments
+ * use the Java types defined in the parameter definitions.
+ * @param columns the set of columns (if any) from the SQL {@code EXTEND} clause
+ *
+ * @return an external table spec which Calcite can consume
+ */
+ protected abstract ExternalTableSpec convertCompletedTable(
Review Comment:
Conversion from a set of parameter values to an external table is a function of the input source and the specific way that we use the table function. We don't want to couple the table function any more tightly to the input source than necessary. This method call is the minimum possible coupling. It also leaves input source logic in the input source.
##########
server/src/main/java/org/apache/druid/catalog/model/table/DatasourceDefn.java:
##########
@@ -19,65 +19,137 @@
package org.apache.druid.catalog.model.table;
+import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
import org.apache.druid.catalog.model.ColumnSpec;
import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.GranularityPropertyDefn;
+import org.apache.druid.catalog.model.ModelProperties.StringListPropertyDefn;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefn;
import org.apache.druid.java.util.common.IAE;
import org.apache.druid.java.util.common.StringUtils;
-import java.util.Collections;
+import java.util.Arrays;
+import java.util.List;
-/**
- * Definition of a Druid datasource. The datasource may use rollup, but rollup
- * is not represented in the catalog: it is just a way that applications store
- * data into a table.
- */
-public class DatasourceDefn extends AbstractDatasourceDefn
+public class DatasourceDefn extends TableDefn
{
/**
- * Definition of a column in a datasource.
+ * Segment grain at ingestion and initial compaction. Aging rules
+ * may override the value as segments age. If not provided here,
+ * then it must be provided at ingestion time.
*/
- public static class DatasourceColumnDefn extends ColumnDefn
- {
- public static final String COLUMN_TYPE = "column";
+ public static final String SEGMENT_GRANULARITY_PROPERTY = "segmentGranularity";
+
+ /**
+ * Catalog property value for the "all time" granularity.
+ */
+ public static final String ALL_GRANULARITY = "ALL";
Review Comment:
It isn't? This is a "property value" not a "property".
##########
server/src/main/java/org/apache/druid/catalog/model/table/InputFormats.java:
##########
@@ -45,59 +42,85 @@
import java.util.stream.Collectors;
/**
- * Definition of the input formats which converts from property
- * lists in table specs to subclasses of {@link InputFormat}.
+ * Catalog definitions for the Druid input formats. Maps from catalog and SQL
+ * functions to the Druid input format implementations.
*/
public class InputFormats
{
- public interface InputFormatDefn
- {
- String name();
- String typeTag();
- List<PropertyDefn<?>> properties();
- void validate(ResolvedTable table);
- InputFormat convert(ResolvedTable table);
- }
-
+ /**
+ * Base class for input format definitions.
+ */
public abstract static class BaseFormatDefn implements InputFormatDefn
{
- private final String name;
- private final String typeTag;
- private final List<PropertyDefn<?>> properties;
-
- public BaseFormatDefn(
- final String name,
- final String typeTag,
- final List<PropertyDefn<?>> properties
- )
+ /**
+ * The set of SQL function parameters available when the format is
+ * specified via a SQL function. The parameters correspond to input format
+ * properties, but typically have simpler names and must require only simple
+ * scalar types of the kind that SQL can provide. Each subclass must perform
+ * conversion to the type required for Jackson conversion.
+ */
+ private final List<ParameterDefn> parameters;
+
+ public BaseFormatDefn(List<ParameterDefn> parameters)
{
- this.name = name;
- this.typeTag = typeTag;
- this.properties = properties;
+ this.parameters = parameters == null ? Collections.emptyList() : parameters;
}
@Override
- public String name()
+ public List<ParameterDefn> parameters()
{
- return name;
+ return parameters;
}
+ /**
+ * The target input format class for Jackson conversions.
+ */
+ protected abstract Class<? extends InputFormat> inputFormatClass();
+
@Override
- public String typeTag()
+ public void validate(ResolvedExternalTable table)
{
- return typeTag;
+ // Bare-bones validation: the format has to convert to the proper object.
+ // Subclasses should replace this with something fancier where needed.
+
+ if (table.inputFormatMap != null) {
+ convertFromTable(table);
+ }
}
- @Override
- public List<PropertyDefn<?>> properties()
+ /**
+ * Convert columns from the {@link ColumnSpec} format used by the catalog to the
+ * list of names form requires by input formats.
+ */
+ protected void convertColumns(Map<String, Object> jsonMap, List<ColumnSpec> columns)
{
- return properties;
+ List<String> cols = columns
+ .stream()
+ .map(col -> col.name())
+ .collect(Collectors.toList());
+ jsonMap.put("columns", cols);
Review Comment:
Sadly, the `csv` format definition includes the list of column names. Rather than requiring the user to provide the list twice (as in the `extern` function), this code "leaches off" of the column definitions used to define the signature so the user doesn't have to repeat themselves.
##########
server/src/main/java/org/apache/druid/catalog/model/table/ExternalTableDefn.java:
##########
@@ -19,254 +19,289 @@
package org.apache.druid.catalog.model.table;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import org.apache.druid.catalog.model.CatalogUtils;
-import org.apache.druid.catalog.model.ColumnDefn;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.druid.catalog.model.ColumnSpec;
-import org.apache.druid.catalog.model.Columns;
-import org.apache.druid.catalog.model.ModelProperties;
+import org.apache.druid.catalog.model.ModelProperties.ObjectPropertyDefn;
import org.apache.druid.catalog.model.ModelProperties.PropertyDefn;
-import org.apache.druid.catalog.model.ParameterizedDefn;
-import org.apache.druid.catalog.model.PropertyAttributes;
import org.apache.druid.catalog.model.ResolvedTable;
import org.apache.druid.catalog.model.TableDefn;
-import org.apache.druid.catalog.model.table.InputFormats.InputFormatDefn;
+import org.apache.druid.catalog.model.TableDefnRegistry;
import org.apache.druid.data.input.InputFormat;
import org.apache.druid.data.input.InputSource;
-import org.apache.druid.java.util.common.IAE;
-import org.apache.druid.java.util.common.ISE;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
+import java.util.Arrays;
import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
/**
- * Definition of an external input source, primarily for ingestion.
+ * Definition of an external table, primarily for ingestion.
* The components are derived from those for Druid ingestion: an
* input source, a format and a set of columns. Also provides
* properties, as do all table definitions.
+ *
+ * <h4>Partial Tables and Connections</h4>
+ *
+ * An input source is a template for an external table. The input
+ * source says how to get data, and optionally the format and structure of that data.
+ * Since Druid never ingests the same data twice, the actual external table needs
+ * details that says which data to read on any specific ingestion. Thus, an external
+ * table is usually a "partial table": all the information that remains constant
+ * across ingestions, but without the information that changes. The changing
+ * information is typically the list of files (or objects or URLs) to ingest.
+ * <p>
+ * The pattern is:<br>
+ * {@code external table spec + parameters --> external table}
+ * <p>
+ * Since an input source is a parameterized (partial) external table, we can reuse
+ * the table metadata structures and APIs, avoiding the need to have a separate (but
+ * otherwise identical) structure for external tables.
+ *
+ * An external table can be thought of as a "connection", though Druid does not
+ * use that term. When used as a connection, the external table spec will omit the
+ * format. Instead, the format will also be provided at ingest time, along with the
+ * list of tables (or objects.)
+ * <p>
+ * To keep all this straight, we adopt the following terms:
+ * <dl>
+ * <dt>External table spec</dt>
+ * <dd>The JSON serialized version of an external table which can be partial or
+ * complete. The spec is a named entry in the Druid catalog</dd>
+ * <dt>Complete spec</dt>
+ * <dd>An external table spec that provides all information needed to access an
+ * external table. Each use identifies the same set of data. Useful if MSQ is used
+ * to query an external data source. A complete spec can be referenced as a
+ * first-class table in a {@code FROM} clause in an MSQ query.</dd>
+ * <dt>Partial spec</dt>
+ * <dd>An external table spec that omits some information. That information must
+ * be provided at query time in the form of a {@code TABLE} function. If the partial
+ * spec includes a format, then it is essentially a <i>partial table</i>. If it
+ * omits the format, then it is essentially a <i>connection</i>.</dd>
+ * <dt>Completed table</dt>
+ * <dd>The full external table that results from a partial spec and a set of SQL
+ * table function parameters.</dd>
+ * <dt>Ad-hoc table</dt>
+ * <dd>Users can define an external table using the generic {@code EXTERN} function
+ * or one of the input-source-specific functions. In this case, there is no
+ * catalog entry: all information comes from the SQL table function</dd>
+ * <dt>Partial table function</dt>
+ * <dd>The SQL table function used to "complete" a partial spec. The function
+ * defines parameters to fill in the missing information. The function is generated
+ * on demand and has the same name as the catalog entry for the partial spec.
+ * The function will include parameters for format if the catalog spec does not
+ * specify a format. Else, the format parameters are omitted and the completed
+ * table uses the format provided in the catalog spec.</dd>
+ * <dt>Ad-hoc table function</dt>
+ * <dd>The SQL table function used to create an ad-hoc external table. The function
+ * as a name defined by the {@link InputFormatDefn}, and has parameters for all
+ * support formats: the user must specify all input source and format properties.</dd>
+ * </dl>
+ *
+ * <h4>External Table Structure</h4>
+ *
+ * The external table is generic: it represents all valid combinations of input
+ * sources and formats. Rather than have a different table definition for each, we
+ * instead split out input sources and formats into their own definitions, and those
+ * definitions are integrated and used by this external table class. As a result,
+ * the {@code properties} field will contain the {@code source} property which has
+ * the JSON serialized form of the input source (minus items to be parameterized.)
+ * <p>
+ * Similarly, if the external table also defines a format (rather than requiring the
+ * format at ingest time), then the {@code format} property holds the JSON-serialized
+ * form of the input format, minus columns. The columns can be provided in the spec,
+ * in the {@code columns} field. The {@link InputFormatDefn} converts the columns to
+ * the form needed by the input format.
+ * <p>
+ * Druid's input sources all require formats. However, some sources may not actually
+ * need the format. A JDBC input source for example, needs no format. In other cases,
+ * there may be a subset of formats. Each {@link InputSourceDefn} is responsible for
+ * working out which formats (if any) are required. This class is agnostic about whether
+ * the format is supplied. (Remember that, when used as a connection, the external table
+ * will provide no format until ingest time.)
+ * <p>
+ * By contrast, the input source is always required.
+ *
+ * <h4>Data Formats and Conversions</h4>
+ *
+ * Much of the code here handles conversion of an external table specification to
+ * the form needed by SQL. Since SQL is not visible here, we instead create an
+ * instance of {@link ExternalTableSpec} which holds the input source, input format
+ * and row signature in the form required by SQL.
+ * <p>
+ * This class handles table specifications in three forms:
+ * <ol>
+ * <li>From a fully-defined table specification, converted to a {@code ExternalTableSpec}
+ * by the {@link #convert(ResolvedTable)} function.</li>
+ * <li>From a fully-defined set of arguments to a SQL table function. The
+ * {@link InputSourceDefn#adHocTableFn()} method provides the function definition which
+ * handles the conversion.</li>
+ * <li>From a partially-defined table specification in the catalog, augmented by
+ * parameters passed from a SQL function. The {@link #tableFn(ResolvedTable)} method
+ * creates the required function by caching the table spec. That function then combines
+ * the parameters to produce the required {@code ExternalTableSpec}.</li>
+ * </ol>
+ * <p>
+ * To handle these formats, and the need to adjust JSON, conversion to an
+ * {@code ExternalTableSpec} occurs in multiple steps:
+ * <ul>
+ * <li>When using a table spec, the serialized JSON is first converted to a generic
+ * Java map: one for the input source, another for the format.</li>
+ * <li>When using a SQL function, the SQL arguments are converted (if needed) and
+ * written into a Java map. If the function references an existing table spec: then
+ * the JSON map is first populated with the deserialized spec.</li>
+ * <li>Validation and/or adjustments are made to the Java map. Adjustments are
+ * those described elsewhere in this Javadoc.</li>
+ * <li>The column specifications from either SQL or the table spec are converted to
+ * a list of column names, and placed into the Java map for the input format.</li>
+ * <li>The maps are converted to the {@link InputSource} or {@link InputFormat}
+ * objects using a Jackson conversion.</li>
+ * </ul>
+ * The actual conversions are handled in the {@link InputFormatDefn} and
+ * {@link InputFormatDefn} classes, either directly (for a fully-defined table
+ * function) or starting here (for other use cases).
+ *
+ * <h4>Property and Parameter Names</h4>
+ *
+ * Pay careful attention to names: the names may be different in each of the above
+ * cases:
+ * <ul>
+ * <li>The table specification stores the input source and input format specs using
+ * the names defined by the classes themselves. That is, the table spec holds a string
+ * that represents the Jackson-serialized form of those classes. In some cases, the
+ * JSON can be a subset: some sources and formats have obscure checks, or options which
+ * are not available via this path. The code that does conversions will adjust the JSON
+ * prior to conversion. Each JSON object has a type field: the value of that type
+ * field must match that defined in the Jackson annotations for the corresponding
+ * class.</li>
+ * <li>SQL table functions use argument names that are typically selected for user
+ * convenience, and may not be the same as the JSON field name. For example, a field
+ * name may be a SQL reserved word, or may be overly long, or may be obscure. The code
+ * for each input source and input format definition does the needed conversion.</li>
+ * <li>Each input source and input format has a type. The input format type is given,
+ * in SQL by the {@code format} property. The format type name is typically the same
+ * as the JSON type name, but need not be.</li>
+ * </ul>
+ *
+ * <h4>Extensions</h4>
+ *
+ * This class is designed to work both with "well known" Druid input sources and formats,
+ * and those defined in an extension. For extension-defined sources and formats to work,
+ * the extension must define an {@link InputSourceDefn} or {@link InputFormatDefn} which
+ * are put into the {@link TableDefnRegistry} and thus available to this class. The result
+ * is that this class is ignorant of the actual details of sources and formats: it instead
+ * delegates to the input source and input format definitions for that work.
* <p>
- * The external table implements the mechanism for parameterized tables,
- * but does not implement the {@link ParameterizedDefn} interface itself.
- * Tables which are parameterized implement that interface to expose
- * methods defined here.
+ * Input sources and input formats defined in an extension are considered "ephemeral":
+ * they can go away if the corresponding extension is removed from the system. In that
+ * case, any table functions defined by those extensions are no longer available, and
+ * any SQL statements that use those functions will no longer work. The catalog may contain
+ * an external table spec that references those definitions. Such specs will continue to
+ * reside in the catalog, and can be retrieved, but they will fail any query that attempts
+ * to reference them.
*/
-public abstract class ExternalTableDefn extends TableDefn
+public class ExternalTableDefn extends TableDefn
{
- public static final String EXTERNAL_COLUMN_TYPE = "extern";
-
- public abstract static class FormattedExternalTableDefn extends ExternalTableDefn
- {
- public static final String FORMAT_PROPERTY = "format";
-
- private final Map<String, InputFormatDefn> formats;
-
- public FormattedExternalTableDefn(
- final String name,
- final String typeValue,
- final List<PropertyDefn<?>> properties,
- final List<ColumnDefn> columnDefns,
- final List<InputFormatDefn> formats
- )
- {
- super(
- name,
- typeValue,
- addFormatProperties(properties, formats),
- columnDefns
- );
- ImmutableMap.Builder<String, InputFormatDefn> builder = ImmutableMap.builder();
- for (InputFormatDefn format : formats) {
- builder.put(format.typeTag(), format);
- }
- this.formats = builder.build();
- }
-
- /**
- * Add format properties to the base set, in the order of the formats,
- * in the order defined by the format. Allow same-named properties across
- * formats, as long as the types are the same.
- */
- private static List<PropertyDefn<?>> addFormatProperties(
- final List<PropertyDefn<?>> properties,
- final List<InputFormatDefn> formats
- )
- {
- List<PropertyDefn<?>> toAdd = new ArrayList<>();
- PropertyDefn<?> formatProp = new ModelProperties.StringPropertyDefn(FORMAT_PROPERTY, PropertyAttributes.SQL_FN_PARAM);
- toAdd.add(formatProp);
- Map<String, PropertyDefn<?>> formatProps = new HashMap<>();
- for (InputFormatDefn format : formats) {
- for (PropertyDefn<?> prop : format.properties()) {
- PropertyDefn<?> existing = formatProps.putIfAbsent(prop.name(), prop);
- if (existing == null) {
- toAdd.add(prop);
- } else if (existing.getClass() != prop.getClass()) {
- throw new ISE(
- "Format %s, property %s of class %s conflicts with another format property of class %s",
- format.name(),
- prop.name(),
- prop.getClass().getSimpleName(),
- existing.getClass().getSimpleName()
- );
- }
- }
- }
- return CatalogUtils.concatLists(properties, toAdd);
- }
-
- @Override
- protected InputFormat convertFormat(ResolvedTable table)
- {
- return formatDefn(table).convert(table);
- }
+ /**
+ * Identifier for external tables.
+ */
+ public static final String TABLE_TYPE = "extern";
- protected InputFormatDefn formatDefn(ResolvedTable table)
- {
- String formatTag = table.stringProperty(FORMAT_PROPERTY);
- if (formatTag == null) {
- throw new IAE("%s property must be set", FORMAT_PROPERTY);
- }
- InputFormatDefn formatDefn = formats.get(formatTag);
- if (formatDefn == null) {
- throw new IAE(
- "Format type [%s] for property %s is not valid",
- formatTag,
- FORMAT_PROPERTY
- );
- }
- return formatDefn;
- }
+ /**
+ * Column type for external tables.
+ */
+ public static final String EXTERNAL_COLUMN_TYPE = "extern";
- @Override
- public void validate(ResolvedTable table)
- {
- super.validate(table);
- formatDefn(table).validate(table);
- List<ColumnSpec> columns = table.spec().columns();
- if (columns == null || columns.isEmpty()) {
- throw new IAE(
- "An external table of type %s must specify one or more columns",
- table.spec().type()
- );
- }
- }
- }
+ /**
+ * Property which holds the input source specification as serialized as JSON.
+ */
+ public static final String SOURCE_PROPERTY = "source";
/**
- * Definition of a column in a detail (non-rollup) datasource.
+ * Property which holds the optional input format specification, serialized as JSON.
*/
- public static class ExternalColumnDefn extends ColumnDefn
- {
- public ExternalColumnDefn()
- {
- super(
- "Column",
- EXTERNAL_COLUMN_TYPE,
- null
- );
- }
+ public static final String FORMAT_PROPERTY = "format";
- @Override
- public void validate(ColumnSpec spec, ObjectMapper jsonMapper)
- {
- super.validate(spec, jsonMapper);
- validateScalarColumn(spec);
- }
- }
+ /**
+ * Definition of the input source property.
+ */
+ private static final PropertyDefn<InputSource> SOURCE_PROPERTY_DEFN =
+ new ObjectPropertyDefn<InputSource>(SOURCE_PROPERTY, InputSource.class);
- protected static final ExternalColumnDefn INPUT_COLUMN_DEFN = new ExternalColumnDefn();
+ /**
+ * Definition of the input format property.
+ */
+ private static final PropertyDefn<InputFormat> FORMAT_PROPERTY_DEFN =
+ new ObjectPropertyDefn<InputFormat>(FORMAT_PROPERTY, InputFormat.class);
- private final List<PropertyDefn<?>> fields;
+ /**
+ * Type reference used to deserialize JSON to a generic map.
+ */
+ @VisibleForTesting
+ public static final TypeReference<Map<String, Object>> MAP_TYPE_REF = new TypeReference<Map<String, Object>>() { };
- public ExternalTableDefn(
- final String name,
- final String typeValue,
- final List<PropertyDefn<?>> fields,
- final List<ColumnDefn> columnDefns
- )
- {
- super(name, typeValue, fields, columnDefns);
- this.fields = fields;
- }
+ private TableDefnRegistry registry;
- public List<PropertyDefn<?>> parameters()
+ public ExternalTableDefn()
{
- return fields.stream()
- .filter(f -> PropertyAttributes.isExternTableParameter(f))
- .collect(Collectors.toList());
+ super(
+ "External table",
+ TABLE_TYPE,
+ Arrays.asList(
+ SOURCE_PROPERTY_DEFN,
+ FORMAT_PROPERTY_DEFN
+ ),
+ null
+ );
}
- public List<PropertyDefn<?>> tableFunctionParameters()
+ @Override
+ public void bind(TableDefnRegistry registry)
{
- return fields.stream()
- .filter(f -> PropertyAttributes.isSqlFunctionParameter(f))
- .collect(Collectors.toList());
+ this.registry = registry;
}
- /**
- * Merge parameters provided by a SQL table function with the catalog information
- * provided in the resolved table to produce a new resolved table used for a
- * specific query.
- */
- public abstract ResolvedTable mergeParameters(ResolvedTable table, Map<String, Object> values);
-
- public ExternalTableSpec convertToExtern(ResolvedTable table)
+ @Override
+ public void validate(ResolvedTable table)
{
- return new ExternalTableSpec(
- convertSource(table),
- convertFormat(table),
- Columns.convertSignature(table.spec())
- );
+ for (PropertyDefn<?> propDefn : properties().values()) {
+ if (propDefn != SOURCE_PROPERTY_DEFN && propDefn != FORMAT_PROPERTY_DEFN) {
Review Comment:
No, because we are comparing instances. We want to match these exact instances, not some other object that happens to be `equals()` to these instances. Recall that most languages support two kinds of identity: instance identity (based on pointers) and valid identity (based, in Java, on `equals()`.) One generally cannot substitute one for the other.
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECTS_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
+ private static final List<ParameterDefn> SECURITY_PARAMS = Arrays.asList(
+ new Parameter(ACCESS_KEY_ID_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(SECRET_ACCESS_KEY_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(ASSUME_ROLE_ARN_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the S3InputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PREFIXES_FIELD = "prefixes";
+ private static final String OBJECTS_FIELD = "objects";
+ private static final String OBJECT_GLOB_FIELD = "objectGlob";
+ private static final String PROPERTIES_FIELD = "properties";
+ private static final String ACCESS_KEY_ID_FIELD = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY_FIELD = "secretAccessKey";
+ private static final String ASSUME_ROLE_ARN_FIELD = "assumeRoleArn";
+
+ @Override
+ public String typeValue()
+ {
+ return S3StorageDruidModule.SCHEME;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return S3InputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (hasFormat && !hasColumns) {
+ throw new IAE(
+ "An external S3 table with a format must also provide the corresponding columns"
+ );
+ }
+
+ // The user can either provide a bucket, or can provide one of the valid items.
+ final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
+ final boolean hasBucket = bucket != null;
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
+ final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
+ final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
+ final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
+ if (hasBucket) {
+ if (hasUris || hasPrefix || hasObjects) {
+ throw new IAE(
+ "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
+ BUCKET_PROPERTY,
+ URIS_FIELD,
+ PREFIXES_FIELD,
+ OBJECTS_FIELD
+ );
+ }
+ if (hasGlob) {
+ throw new IAE(
+ "The %s property cannot be provided when the the %s property is set",
+ OBJECT_GLOB_FIELD,
+ BUCKET_PROPERTY
+ );
+ }
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ sourceMap.put(URIS_FIELD, Collections.singletonList(bucket));
Review Comment:
IIRC, the S3 input source requires one of a URIs, objects or some such. (Druid does not support an S3 input source with nothing to read.)
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ return new ExternalTableSpec(
+ convertArgsToSource(args, jsonMapper),
+ convertArgsToFormat(args, columns, jsonMapper),
+ Columns.convertSignature(columns)
+ );
+ }
+
+ /**
+ * Convert the input source using arguments to a "from scratch" table function.
+ */
+ protected InputSource convertArgsToSource(Map<String, Object> args, ObjectMapper jsonMapper)
Review Comment:
See note above. Table functions are generic. Here we're using them for input sources.
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
This is an error message, copied from somewhere in MSQ. The error message shouldn't get copied into the spec (unless I've done something very wrong.)
There was a comment in an early PR that someone didn't like two ways to specify granularities, and we wanted only the ISO 8601 format. So, the user should use that. Easy enough to fix if we want to do something else.
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
Review Comment:
The idea is that the table function is generic. Here, the base input source says that, if you don't want to do anything different _for input sources_, just do the generic conversion. If this moved into the table function class, it would make the table function class only applicable to input sources, and would require binding the base table function class to an input source. The present design avoids this unwanted coupling.
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ protected final Map<String, Object> inputSourceMap;
+ protected final Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ map = table.mapProperty(ExternalTableDefn.FORMAT_PROPERTY);
+ this.inputFormatMap = map == null ? null : new HashMap<>(map);
+ }
+
+ public ResolvedTable resolvedTable()
+ {
+ return table;
+ }
+
+ /**
+ * Look up the input source type and input format type to find the corresponding
+ * definitions in the table registry. Throws an exception if the types are not
+ * defined. The input source is required, the format is optional.
+ * <p>
+ * Note, for resolution to work, the name of each definition must be the same as
+ * that used as the type key in the serialized JSON.
+ */
+ public ResolvedExternalTable resolve(TableDefnRegistry registry)
+ {
+ String inputSourceType = CatalogUtils.getString(inputSourceMap, InputSource.TYPE_PROPERTY);
+ if (inputSourceType == null) {
+ throw new IAE("Input source type %s is required", InputSource.TYPE_PROPERTY);
+ }
+ inputSourceDefn = registry.inputSourceDefnFor(inputSourceType);
+ if (inputSourceDefn == null) {
+ throw new IAE("Input source type %s is not registered", inputSourceType);
+ }
+ if (inputFormatMap != null) {
+ String inputFormatType = CatalogUtils.getString(inputFormatMap, InputFormat.TYPE_PROPERTY);
+ if (inputFormatType != null) {
Review Comment:
This is a simple holder for information read from the catalog. It is perfectly legal for the catalog to define an external table that has no input format. Imagine a "connection" kind of thing: I specify to use S3, provide the bucket, provide credentials, etc. But, I expect each query to provide the actual file names and the format of those files. In that case, the catalog entry for my "connection" would contain no format: the code here will require, however, that I provide that format via a table function at query time.
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefnRegistry.java:
##########
@@ -55,40 +61,76 @@
{
// Temporary list of Druid-define table definitions. This should come from
// Guice later to allow extensions to define table types.
- private static final TableDefn[] TABLE_DEFNS = {
+ private static final List<TableDefn> BUILTIN_TABLE_DEFNS = Arrays.asList(
new DatasourceDefn(),
- new InlineTableDefn(),
- new HttpTableDefn(),
- new LocalTableDefn()
- };
+ new ExternalTableDefn()
+ );
+ private static final List<InputSourceDefn> BUILTIN_INPUT_SOURCE_DEFNS = Arrays.asList(
+ new InlineInputSourceDefn(),
+ new HttpInputSourceDefn(),
+ new LocalInputSourceDefn()
+ );
+ private static final List<InputFormatDefn> BUILTIN_INPUT_FORMAT_DEFNS = Arrays.asList(
+ new InputFormats.CsvFormatDefn(),
+ new InputFormats.DelimitedFormatDefn(),
+ new InputFormats.JsonFormatDefn()
+ );
- private final Map<String, TableDefn> defns;
+ private final Map<String, TableDefn> tableDefns;
+ private final Map<String, InputSourceDefn> inputSourceDefns;
+ private final Map<String, InputFormatDefn> inputFormatDefns;
private final ObjectMapper jsonMapper;
public TableDefnRegistry(
- final TableDefn[] defns,
+ @Nullable final List<TableDefn> tableDefnExtns,
+ @Nullable final List<InputSourceDefn> inputSourceDefnExtns,
+ @Nullable final List<InputFormatDefn> inputFormatDefnExtns,
final ObjectMapper jsonMapper
)
{
- ImmutableMap.Builder<String, TableDefn> builder = ImmutableMap.builder();
- for (TableDefn defn : defns) {
- builder.put(defn.typeValue(), defn);
- }
- this.defns = builder.build();
this.jsonMapper = jsonMapper;
+ final List<TableDefn> tableDefns = CatalogUtils.concatLists(tableDefnExtns, BUILTIN_TABLE_DEFNS);
+ final List<InputSourceDefn> inputSourceDefns = CatalogUtils.concatLists(inputSourceDefnExtns, BUILTIN_INPUT_SOURCE_DEFNS);
+ final List<InputFormatDefn> inputFormatDefns = CatalogUtils.concatLists(inputFormatDefnExtns, BUILTIN_INPUT_FORMAT_DEFNS);
+
+ ImmutableMap.Builder<String, TableDefn> tableBuilder = ImmutableMap.builder();
+ for (TableDefn defn : tableDefns) {
+ tableBuilder.put(defn.typeValue(), defn);
+ }
+ this.tableDefns = tableBuilder.build();
+
+ ImmutableMap.Builder<String, InputSourceDefn> sourceBuilder = ImmutableMap.builder();
+ for (InputSourceDefn defn : inputSourceDefns) {
+ sourceBuilder.put(defn.typeValue(), defn);
+ }
+ this.inputSourceDefns = sourceBuilder.build();
+
+ ImmutableMap.Builder<String, InputFormatDefn> formatBuilder = ImmutableMap.builder();
+ for (InputFormatDefn defn : inputFormatDefns) {
+ formatBuilder.put(defn.typeValue(), defn);
+ }
+ this.inputFormatDefns = formatBuilder.build();
+
+ // Initialize all items once the entire set of bindings is defined.
+ for (InputSourceDefn defn : inputSourceDefns) {
+ defn.bind(this);
+ }
+ for (TableDefn defn : tableDefns) {
+ defn.bind(this);
+ }
Review Comment:
We could, but, at present, input formats don't depend on anything else. Input source definitions depend on input sources. External tables depend on input sources. But formats are at the leaf: they don't depend on anything.
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ protected final Map<String, Object> inputSourceMap;
+ protected final Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ map = table.mapProperty(ExternalTableDefn.FORMAT_PROPERTY);
+ this.inputFormatMap = map == null ? null : new HashMap<>(map);
+ }
+
+ public ResolvedTable resolvedTable()
+ {
+ return table;
+ }
+
+ /**
+ * Look up the input source type and input format type to find the corresponding
+ * definitions in the table registry. Throws an exception if the types are not
+ * defined. The input source is required, the format is optional.
+ * <p>
+ * Note, for resolution to work, the name of each definition must be the same as
+ * that used as the type key in the serialized JSON.
+ */
+ public ResolvedExternalTable resolve(TableDefnRegistry registry)
+ {
+ String inputSourceType = CatalogUtils.getString(inputSourceMap, InputSource.TYPE_PROPERTY);
+ if (inputSourceType == null) {
+ throw new IAE("Input source type %s is required", InputSource.TYPE_PROPERTY);
+ }
+ inputSourceDefn = registry.inputSourceDefnFor(inputSourceType);
+ if (inputSourceDefn == null) {
+ throw new IAE("Input source type %s is not registered", inputSourceType);
+ }
+ if (inputFormatMap != null) {
+ String inputFormatType = CatalogUtils.getString(inputFormatMap, InputFormat.TYPE_PROPERTY);
+ if (inputFormatType != null) {
+ inputFormatDefn = registry.inputFormatDefnFor(inputFormatType);
+ if (inputFormatDefn == null) {
+ throw new IAE("Input format type %s is not registered", inputFormatType);
+ }
+ }
+ }
+ return this;
+ }
+
+ /**
+ * Validate that the table spec is correct by resolving the definitions, then
+ * converting the JSON to the desired object type. Note that this path requires
+ * special handling: the table spec may be partial, which means it is missing information
+ * needed to create a complete input source. The input source definition defines which
+ * values can be omitted, and defined later in SQL via function parameters. If those
+ * values are missing, then the input source defn should provide dummy values so that
Review Comment:
The catalog can hold a partial input source. See above comment for a use case. We still want to validate the properties that are provided. Druid input sources are designed to work only if all properties are provided: they don't "do" partial definitions. To avoid the need to duplicate existing code, and to allow the catalog to support partition definitions, even if Druid doesn't, we just patch in dummy missing values to keep Druid happy. Ugly? Of course. But, easier than trying to convince Druid's input sources to change.
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseTableFunction.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.List;
+
+/**
+ * Base implementation for a table function definition.
+ *
+ * @see {@link TableFunction}
+ */
+public abstract class BaseTableFunction implements TableFunction
+{
+ public static class Parameter implements ParameterDefn
+ {
+ private final String name;
+ private final ParameterType type;
+ private final boolean optional;
+
+ public Parameter(String name, ParameterType type, boolean optional)
+ {
+ this.name = name;
+ this.type = type;
+ this.optional = optional;
+ }
+
+ @Override
+ public String name()
+ {
+ return name;
+ }
+
+ @Override
+ public ParameterType type()
+ {
+ return type;
+ }
+
+ @Override
+ public boolean isOptional()
+ {
+ return optional;
+ }
+ }
+
+ private final List<ParameterDefn> parameters;
+
+ public BaseTableFunction(List<ParameterDefn> parameters)
+ {
+ this.parameters = parameters;
+ }
+
+ @Override
+ public List<ParameterDefn> parameters()
+ {
+ return parameters;
+ }
+
+ protected void requireSchema(String fnName, List<ColumnSpec> columns)
+ {
+ if (columns == null) {
Review Comment:
Most databases allow tables with zero columns. Druid isn't a DB: it is a tool for specific kinds of apps. Those apps don't allow inputs with zero columns. So, to avoid blowing up later, we catch the problem as early as possible.
More realistically, someone didn't realize that they have to tell MSQ the input schema, and just forgot to include it.
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidCalciteSchemaModule.java:
##########
@@ -64,7 +64,7 @@ public void configure(Binder binder)
}
@Provides
- @Singleton
+ @LazySingleton
Review Comment:
@imply-cheddar, thanks for your strident note. As it turns out, we can't _quite_ achieve what you ask. There is one other use of `@Singleton` in the code base, which I added in a previous PR. It handles the case in which Guice creates an instance explicitly. You indicated that you wanted only explicit bindings, but had to turn off that protection for some reason or other. The result is that, when Guice is used in tests, it will occasionally find itself implicitly instantiating some instances.
There are times when we need those implicit instances to be singletons. These implicit singletons are where the `@Singleton` or `@LazySingleton` tags are needed. (Again, the binding is "implicit" if there is no explicit binding, which means we didn't use `@LazySingleton` at that time.)
This is an example of a class that sometimes gets instantiated implicitly. So, we have to mark the class as `@[Lazy]Singleton`.
There is one other place in the code where changing `@Singleton` to `@LazySingleton` cases Guice to crash. Why? I didn't take the time to figure it out. That line _didn't_ change in this PR, but is the one we should be discussing.
To achieve @imply-cheddar 's dictate, we have to turn off implicit instantiation. By so doing, we eliminate the need to annotate classes.
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/Externals.java:
##########
@@ -145,49 +174,41 @@ public Consistency getConsistency()
}
/**
- * Convert the actual arguments to SQL external table function into a catalog
- * resolved table, then convert that to an external table spec usable by MSQ.
- *
- * @param tableDefn catalog definition of the kind of external table
- * @param parameters the parameters to the SQL table macro
- * @param arguments the arguments that match the parameters. Optional arguments
- * may be null
- * @param schema the external table schema provided by the EXTEND clause
- * @param jsonMapper the JSON mapper to use for value conversions
- * @return a spec with the three values that MSQ needs to create an external table
+ * Convert the list of Calcite function arguments to a map of non-null arguments.
+ * The resulting map must be mutable as processing may rewrite values.
*/
- public static ExternalTableSpec convertArguments(
- final ExternalTableDefn tableDefn,
- final List<FunctionParameter> parameters,
- final List<Object> arguments,
- final SqlNodeList schema,
- final ObjectMapper jsonMapper
+ public static Map<String, Object> convertArguments(
+ final TableFunction fn,
+ final List<Object> arguments
)
{
- final TableBuilder builder = TableBuilder.of(tableDefn);
- for (int i = 0; i < parameters.size(); i++) {
- String name = parameters.get(i).getName();
- Object value = arguments.get(i);
- if (value == null) {
- continue;
+ final List<TableFunction.ParameterDefn> params = fn.parameters();
+ final Map<String, Object> argMap = new HashMap<>();
+ for (int i = 0; i < arguments.size(); i++) {
+ final Object value = arguments.get(i);
+ if (value != null) {
+ argMap.put(params.get(i).name(), value);
}
- PropertyDefn<?> prop = tableDefn.property(name);
- builder.property(name, prop.decodeSqlValue(value, jsonMapper));
}
+ return argMap;
+ }
- // Converts from a list of (identifier, type, ...) pairs to
- // a Druid row signature. The schema itself comes from the
- // Druid-specific EXTEND syntax added to the parser.
+ /**
+ * Converts from a list of (identifier, type, ...) pairs to
+ * list of column specs. The schema itself comes from the
+ * Druid-specific EXTEND syntax added to the parser.
+ */
+ public static List<ColumnSpec> convertColumns(SqlNodeList schema)
+ {
+ final List<ColumnSpec> columns = new ArrayList<>();
for (int i = 0; i < schema.size(); i += 2) {
final String name = convertName((SqlIdentifier) schema.get(i));
- String sqlType = convertType(name, (SqlDataTypeSpec) schema.get(i + 1));
- builder.column(name, sqlType);
+ final String sqlType = convertType(name, (SqlDataTypeSpec) schema.get(i + 1));
+ columns.add(new ColumnSpec(name, sqlType, null));
Review Comment:
We're in code that is processing the _input schema_ for an external table. This is not code for handling a data source table. MSQ requires that an input schema MUST define the column type. If the type was not given, we should have failed validation at some earlier point as the syntax would not have been valid.
By asking about missing null column types, you may be asking about the behavior of columns defined for a data source, which is a topic handled elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1069786027
##########
extensions-core/s3-extensions/pom.xml:
##########
@@ -114,7 +120,6 @@
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
- <version>${aws.sdk.version}</version>
Review Comment:
Yes. We have the following in the root `pom.xml`:
```xml
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
```
When we manage versions at the root level, it is redundant (and error-prone) to repeat the version in child POMs. My IDE gives a warning when we do so. This change acknowledges that we manage versions for this artifact at the root level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
abhishekrb19 commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067732431
##########
server/src/main/java/org/apache/druid/catalog/model/table/InlineInputSourceDefn.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.InlineInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Describes an inline input source: one where the data is provided in the
+ * table spec as a series of text lines. Since the data is provided, the input
+ * format is required in the table spec: it cannot be provided at ingest time.
+ * Primarily for testing.
+ */
+public class InlineInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = InlineInputSource.TYPE_KEY;
+ public static final String DATA_PROPERTY = "data";
+
+ @Override
+ public String typeValue()
+ {
+ return TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return InlineInputSource.class;
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return Collections.singletonList(
+ new Parameter(DATA_PROPERTY, ParameterType.VARCHAR_ARRAY, false)
+ );
+ }
+
+ @Override
+ public TableFunction partialTableFn(ResolvedExternalTable table)
+ {
+ return new PartialTableFunction(table, Collections.emptyList());
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ // For inline, format is required to match the data
+ if (table.inputFormatMap == null) {
+ throw new IAE("An inline input source must provide a format.");
+ }
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ throw new IAE("An inline input source must provide one or more columns");
+ }
+
+ super.validate(table);
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, InlineInputSource.TYPE_KEY);
+ List<String> data = CatalogUtils.getStringArray(args, DATA_PROPERTY);
+
+ // Would be nice, from a completeness perspective, for the inline data
+ // source to allow zero rows of data. However, such is not the case.
+ if (CollectionUtils.isNullOrEmpty(data)) {
+ throw new IAE(
+ "An inline table requires one or more rows of data in the '%s' property",
+ DATA_PROPERTY
+ );
+ }
+ jsonMap.put("data", String.join("\n", data));
+ }
+
+ @Override
+ protected ExternalTableSpec convertCompletedTable(
+ final ResolvedExternalTable table,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns
+ )
+ {
+ if (!args.isEmpty()) {
+ throw new ISE("Cannot provide arguments for an inline table");
+ }
+ if (!CollectionUtils.isNullOrEmpty(columns)) {
+ throw new IAE("Cannot provide columns for an inline table");
+ }
+ return convertTable(table);
+ }
+
+ @Override
+ protected void auditInputSource(Map<String, Object> jsonMap)
+ {
+ // Special handling of the data property which, in SQL, is a null-delimited
+ // list of rows. The user will usually provide a trailing newline which should
+ // not be interpreted as an empty data row. That is, if the data ends with
+ // a newline, the inline input source will interpret that as a blank line, oddly.
+ String data = CatalogUtils.getString(jsonMap, "data");
+ if (data != null && !data.endsWith("\n")) {
+ jsonMap.put("data", data + "\n");
Review Comment:
```suggestion
jsonMap.put(DATA_PROPERTY, data + "\n");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] imply-cheddar commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1069039794
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidCalciteSchemaModule.java:
##########
@@ -64,7 +64,7 @@ public void configure(Binder binder)
}
@Provides
- @Singleton
+ @LazySingleton
Review Comment:
Please share where that LazySingleton thing is breaking? It should be true that `LazySingleton` is the only singleton annotation that we use anywhere, if that's breaking for a reason, I'm very curious how it's breaking stuff and would like to help fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066584727
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ return new ExternalTableSpec(
+ convertArgsToSource(args, jsonMapper),
+ convertArgsToFormat(args, columns, jsonMapper),
+ Columns.convertSignature(columns)
+ );
+ }
+
+ /**
+ * Convert the input source using arguments to a "from scratch" table function.
+ */
+ protected InputSource convertArgsToSource(Map<String, Object> args, ObjectMapper jsonMapper)
Review Comment:
This this be moved into the AdHocTableFunction class, since it seems it should only be used in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066576021
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseTableFunction.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.List;
+
+/**
+ * Base implementation for a table function definition.
+ *
+ * @see {@link TableFunction}
+ */
+public abstract class BaseTableFunction implements TableFunction
+{
+ public static class Parameter implements ParameterDefn
+ {
+ private final String name;
+ private final ParameterType type;
+ private final boolean optional;
+
+ public Parameter(String name, ParameterType type, boolean optional)
+ {
+ this.name = name;
+ this.type = type;
+ this.optional = optional;
+ }
+
+ @Override
+ public String name()
+ {
+ return name;
+ }
+
+ @Override
+ public ParameterType type()
+ {
+ return type;
+ }
+
+ @Override
+ public boolean isOptional()
+ {
+ return optional;
+ }
+ }
+
+ private final List<ParameterDefn> parameters;
+
+ public BaseTableFunction(List<ParameterDefn> parameters)
+ {
+ this.parameters = parameters;
+ }
+
+ @Override
+ public List<ParameterDefn> parameters()
+ {
+ return parameters;
+ }
+
+ protected void requireSchema(String fnName, List<ColumnSpec> columns)
+ {
+ if (columns == null) {
Review Comment:
Should this also fail if the list of columns is empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067627186
##########
server/src/main/java/org/apache/druid/catalog/model/table/LocalInputSourceDefn.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.LocalInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Definition for a {@link LocalInputSource}.
+ */
+public class LocalInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = LocalInputSource.TYPE_KEY;
+
+ /**
+ * Base directory for file or filter operations. If not provided,
+ * then the servers current working directory is assumed, which is
+ * typically valid only for sample data.
+ */
+ public static final String BASE_DIR_PARAMETER = "baseDir";
+
+ // Note name "fileFilter", not "filter". These properties mix in with
+ // others and "filter" is a bit too generic in that context.
+ public static final String FILTER_PARAMETER = "filter";
+ public static final String FILES_PARAMETER = "files";
+
+ protected static final String BASE_DIR_FIELD = "baseDir";
+ protected static final String FILES_FIELD = "files";
+ protected static final String FILTER_FIELD = "filter";
+
+ private static final ParameterDefn FILTER_PARAM_DEFN =
+ new Parameter(FILTER_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn FILES_PARAM_DEFN =
+ new Parameter(FILES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+
+ @Override
+ public String typeValue()
+ {
+ return LocalInputSource.TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return LocalInputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasBaseDir = sourceMap.containsKey(BASE_DIR_FIELD);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ final boolean hasFilter = !Strings.isNullOrEmpty(CatalogUtils.getString(sourceMap, FILTER_FIELD));
+
+ if (!hasBaseDir && !hasFiles) {
+ throw new IAE(
+ "A local input source requires one property of %s or %s",
+ BASE_DIR_FIELD,
+ FILES_FIELD
+ );
+ }
+ if (!hasBaseDir && hasFilter) {
+ throw new IAE(
+ "If a local input source sets property %s, it must also set property %s",
+ FILTER_FIELD,
+ BASE_DIR_FIELD
+ );
+ }
+ if (hasBaseDir && hasFiles) {
+ throw new IAE(
+ "A local input source accepts only one of %s or %s",
+ BASE_DIR_FIELD,
+ FILES_FIELD
+ );
+ }
+ super.validate(table);
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return Arrays.asList(
+ new Parameter(BASE_DIR_PARAMETER, ParameterType.VARCHAR, true),
+ FILTER_PARAM_DEFN,
+ FILES_PARAM_DEFN
+ );
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, LocalInputSource.TYPE_KEY);
+
+ final String baseDirParam = CatalogUtils.getString(args, BASE_DIR_PARAMETER);
+ final List<String> filesParam = CatalogUtils.getStringArray(args, FILES_PARAMETER);
+ final String filterParam = CatalogUtils.getString(args, FILTER_PARAMETER);
+ final boolean hasBaseDir = !Strings.isNullOrEmpty(baseDirParam);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(filesParam);
+ final boolean hasFilter = !Strings.isNullOrEmpty(filterParam);
+ if (!hasBaseDir && !hasFiles) {
+ throw new IAE(
+ "A local input source requires one parameter of %s or %s",
+ BASE_DIR_PARAMETER,
+ FILES_PARAMETER
+ );
+ }
+ if (hasBaseDir && hasFiles) {
+ if (hasFilter) {
+ throw new IAE(
+ "A local input source can set parameter %s or %s, but not both.",
+ FILES_PARAMETER,
+ FILTER_PARAMETER
+ );
+ }
+ // Special workaround: if the user provides a base dir, and files, convert
+ // the files to be absolute paths relative to the base dir. Then, remove
+ // the baseDir, since the Druid input source does not allow both a baseDir
+ // and a list of files.
+ jsonMap.put(FILES_FIELD, absolutePath(baseDirParam, filesParam));
+ return;
+ }
+ if (!hasBaseDir && !Strings.isNullOrEmpty(filterParam)) {
+ throw new IAE(
+ "If a local input source sets parameter %s, it must also set parameter %s",
+ FILTER_PARAMETER,
+ BASE_DIR_PARAMETER
+ );
+ }
+ if (hasBaseDir) {
+ jsonMap.put(BASE_DIR_FIELD, baseDirParam);
+ }
+ if (hasFiles) {
+ jsonMap.put(FILES_FIELD, filesParam);
+ }
+ if (filterParam != null) {
+ jsonMap.put(FILTER_FIELD, filterParam);
+ }
+ }
+
+ private List<String> absolutePath(String baseDirPath, List<String> files)
+ {
+ final File baseDir = new File(baseDirPath);
+ return files.stream()
+ .map(f -> new File(baseDir, f).toString())
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public TableFunction partialTableFn(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ final boolean hasFilter = !Strings.isNullOrEmpty(CatalogUtils.getString(sourceMap, FILTER_FIELD));
+ List<ParameterDefn> params = new ArrayList<>();
+ if (!hasFiles && !hasFilter) {
+ params.add(FILES_PARAM_DEFN);
+ params.add(FILTER_PARAM_DEFN);
+ }
+
+ // Does the table define a format?
+ if (table.inputFormatMap == null) {
+ params = addFormatParameters(params);
+ }
+ return new PartialTableFunction(table, params);
+ }
+
+ @Override
+ protected ExternalTableSpec convertCompletedTable(
+ final ResolvedExternalTable table,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns
+ )
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ if (hasFiles) {
+ if (!args.isEmpty()) {
+ throw new IAE("The local input source has a file list: do not provide other arguments");
+ }
+ } else {
+ final String baseDir = CatalogUtils.getString(sourceMap, BASE_DIR_FIELD);
+ if (Strings.isNullOrEmpty(baseDir)) {
+ throw new IAE(
+ "When an inline external table is used with a table function, %s must be set",
Review Comment:
why is this error message talking about inline external table? Isnt this a Local input source?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on PR #13627:
URL: https://github.com/apache/druid/pull/13627#issuecomment-1386246212
@maytasm, thanks for checking. @zachjsh had given his approval, but I see it wasn't actually recorded here. @zachjsh, can you do the honors? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067641161
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ protected final Map<String, Object> inputSourceMap;
+ protected final Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ map = table.mapProperty(ExternalTableDefn.FORMAT_PROPERTY);
+ this.inputFormatMap = map == null ? null : new HashMap<>(map);
+ }
+
+ public ResolvedTable resolvedTable()
+ {
+ return table;
+ }
+
+ /**
+ * Look up the input source type and input format type to find the corresponding
+ * definitions in the table registry. Throws an exception if the types are not
+ * defined. The input source is required, the format is optional.
+ * <p>
+ * Note, for resolution to work, the name of each definition must be the same as
+ * that used as the type key in the serialized JSON.
+ */
+ public ResolvedExternalTable resolve(TableDefnRegistry registry)
+ {
+ String inputSourceType = CatalogUtils.getString(inputSourceMap, InputSource.TYPE_PROPERTY);
+ if (inputSourceType == null) {
+ throw new IAE("Input source type %s is required", InputSource.TYPE_PROPERTY);
+ }
+ inputSourceDefn = registry.inputSourceDefnFor(inputSourceType);
+ if (inputSourceDefn == null) {
+ throw new IAE("Input source type %s is not registered", inputSourceType);
+ }
+ if (inputFormatMap != null) {
+ String inputFormatType = CatalogUtils.getString(inputFormatMap, InputFormat.TYPE_PROPERTY);
+ if (inputFormatType != null) {
+ inputFormatDefn = registry.inputFormatDefnFor(inputFormatType);
+ if (inputFormatDefn == null) {
+ throw new IAE("Input format type %s is not registered", inputFormatType);
+ }
+ }
+ }
+ return this;
+ }
+
+ /**
+ * Validate that the table spec is correct by resolving the definitions, then
+ * converting the JSON to the desired object type. Note that this path requires
+ * special handling: the table spec may be partial, which means it is missing information
+ * needed to create a complete input source. The input source definition defines which
+ * values can be omitted, and defined later in SQL via function parameters. If those
+ * values are missing, then the input source defn should provide dummy values so that
Review Comment:
question: Where are these dummy values that I've seen set in places removed from the definition of the input source, so that we dont potentially run an ingestion with dummy values set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
abhishekrb19 commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067766205
##########
server/src/main/java/org/apache/druid/catalog/model/table/LocalInputSourceDefn.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.LocalInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Definition for a {@link LocalInputSource}.
+ */
+public class LocalInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = LocalInputSource.TYPE_KEY;
+
+ /**
+ * Base directory for file or filter operations. If not provided,
+ * then the servers current working directory is assumed, which is
+ * typically valid only for sample data.
+ */
+ public static final String BASE_DIR_PARAMETER = "baseDir";
+
+ // Note name "fileFilter", not "filter". These properties mix in with
Review Comment:
Is this comment stale? Asking because the property below is named "filter", not "fileFilter"
##########
extensions-core/s3-extensions/pom.xml:
##########
@@ -114,7 +120,6 @@
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
- <version>${aws.sdk.version}</version>
Review Comment:
Is this change intentional?
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ /**
+ * External data table spec property that let's the user define one bucket in the catalog,
+ * so that the corresponding table function can supply just the relative path names within
+ * that bucket. That is, if the user sets this, Druid will generate the {@code objects}
+ * field from this entry and files provided in the table function.
+ */
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECT_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
+ private static final List<ParameterDefn> SECURITY_PARAMS = Arrays.asList(
+ new Parameter(ACCESS_KEY_ID_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(SECRET_ACCESS_KEY_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(ASSUME_ROLE_ARN_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the S3InputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PREFIXES_FIELD = "prefixes";
+ private static final String OBJECTS_FIELD = "objects";
+ private static final String OBJECT_GLOB_FIELD = "objectGlob";
+ private static final String PROPERTIES_FIELD = "properties";
+ private static final String ACCESS_KEY_ID_FIELD = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY_FIELD = "secretAccessKey";
+ private static final String ASSUME_ROLE_ARN_FIELD = "assumeRoleArn";
+
+ @Override
+ public String typeValue()
+ {
+ return S3StorageDruidModule.SCHEME;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return S3InputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (hasFormat && !hasColumns) {
+ throw new IAE(
+ "An external S3 table with a format must also provide the corresponding columns"
+ );
+ }
+
+ // The user can either provide a bucket, or can provide one of the valid items.
+ final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
+ final boolean hasBucket = bucket != null;
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
+ final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
+ final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
+ final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
+ if (hasBucket) {
+ if (hasUris || hasPrefix || hasObjects) {
+ throw new IAE(
+ "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
+ BUCKET_PROPERTY,
+ URIS_FIELD,
+ PREFIXES_FIELD,
+ OBJECTS_FIELD
+ );
+ }
+ if (hasGlob) {
+ throw new IAE(
+ "The %s property cannot be provided when the the %s property is set",
Review Comment:
nit: repeated "the" twice
```suggestion
"The %s property cannot be provided when the %s property is set",
```
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ /**
+ * External data table spec property that let's the user define one bucket in the catalog,
Review Comment:
nit: let's -> lets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on pull request #13627: Improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on PR #13627:
URL: https://github.com/apache/druid/pull/13627#issuecomment-1386467104
@paul-rogers , hmm it doesnt seem to give me the ability to approve the pr anymore 😢 . But I do approve of change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066147525
##########
docs/multi-stage-query/reference.md:
##########
@@ -88,93 +118,260 @@ FROM TABLE(
http(
userName => 'bob',
password => 'secret',
- uris => 'http:foo.com/bar.csv',
+ uris => ARRAY['http:example.com/foo.csv', 'http:example.com/bar.csv'],
format => 'csv'
)
) (x VARCHAR, y VARCHAR, z BIGINT)
```
+#### Function Arguments
-The set of table functions and formats is preliminary in this release.
+These table functions are intended for use with the SQL by-name argument syntax
+as shown above. Because the functions include all parameters for all formats,
+using positional calls is both cumbersome and error-prone.
+
+Function argument names are generally the same as the JSON field names, except
+as noted below. Each argument has a SQL type which matches the JSON type. For
+arguments that take a string list in JSON, use the SQL `ARRAY[...]` syntax in
+SQL as shown in the above example.
+
+Array parameters are good candidates for use in parameterized queries. That is:
+
+```sql
+SELECT
+ <column>
+FROM TABLE(
+ http(
+ userName => 'bob',
+ password => 'secret',
+ uris => ?,
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR, z BIGINT)
+```
-#### `HTTP`
+Provide the list of URIs (in this case) as a query parameter in each ingest. Doing
+so is simpler than writing a script to insert the array into the SQL text.
-The `HTTP` table function represents the `HttpInputSource` class in Druid which allows you to
-read from an HTTP server. The function accepts the following arguments:
+#### `HTTP` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `userName` | Basic authentication user name | `httpAuthenticationUsername` | No |
-| `password` | Basic authentication password | `httpAuthenticationPassword` | No |
-| `passwordEnvVar` | Environment variable that contains the basic authentication password| `httpAuthenticationPassword` | No |
-| `uris` | Comma-separated list of URIs to read. | `uris` | Yes |
+The `HTTP` table function represents the
+[HTTP input source](../ingestion/native-batch-input-sources.md#http-input-source)
+to read from an HTTP server. The function accepts the following arguments:
-#### `INLINE`
+* `userName` (`VARCHAR`) - Same as JSON `httpAuthenticationUsername`.
+* `password` (`VARCHAR`) - Same as`httpAuthenticationPassword` when used with the default option.
+* `passwordEnvVar` (`VARCHAR`) - Same as the HTTP `httpAuthenticationPassword` when used with
+ the `"type": "environment"` option.
+* `uris` (`ARRAY` of `VARCHAR`)
-The `INLINE` table function represents the `InlineInputSource` class in Druid which provides
-data directly in the table function. The function accepts the following arguments:
+#### `INLINE` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `data` | Text lines of inline data. Separate lines with a newline. | `data` | Yes |
+The `INLINE` table function represents the
+[Inline input source](../ingestion/native-batch-input-sources.md#inline-input-source)
+which provides data directly in the table function. Parameter:
-#### `LOCALFILES`
+* `data` (`ARRAY` of `VARCHAR`) - Data lines, without a trailing newline, as an array.
-The `LOCALFILES` table function represents the `LocalInputSource` class in Druid which reads
+Example:
+
+```sql
+FROM TABLE(
+ inline(
+ data => ARRAY[
+ "a,b",
+ "c,d"],
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR)
+```
+
+
+#### `LOCALFILES` Function
+
+The `LOCALFILES` table function represents the
+[Local input source](../ingestion/native-batch-input-sources.md#local-input-source) which reads
files from the file system of the node running Druid. This is most useful for single-node
-installations. The function accepts the following arguments:
+installations. The function accepts the following parameters:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `baseDir` | Directory to read from. | `baseDir` | No |
-| `filter` | Filter pattern to read. Example: `*.csv`. | `filter` | No |
-| `files` | Comma-separated list of files to read. | `files` | No |
+* `baseDir`
+* `filter`
+* `files`
-You must either provide the `baseDir` or the list of `files`. You can provide both, in which case
-the files are assumed relative to the `baseDir`. If you provide a `filter`, you must provide the
-`baseDir`.
+When the local files input source is used directly in an `extern` function, or ingestion spec, you
+can provide either `baseDir` and `filter` or `files` but not both. This function, however, allows
+you to provide any of the following combinations:
-#### Table Function Format
+* `baseDir` - Matches all files in the given directory. (Assumes the filter is `*`.)
+* `baseDir` and `filter` - Match files in the given directory using the filter.
+* `baseDir` and `files` - A set of files relative to `baseDir`.
+* `files` - The files should be absolute paths, else they will be computed relative to Druid's
+ working directory (usually the Druid install directory.)
-Each of the table functions above requires that you specify a format.
+Examples:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `format` | The input format, using the same names as for `EXTERN`. | `inputFormat.type` | Yes |
+```sql
+ -- All files in /tmp, which must be CSV files
+ localfiles(baseDir => '/tmp',
+ format => 'csv')
+
+ -- CSV files in /tmp
+ localfiles(baseDir => '/tmp',
+ filter => '*.csv',
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(baseDir => '/tmp',
+ files => ARRAY['a.csv', 'b.csv'],
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(files => ARRAY['/tmp/a.csv', '/tmp/b.csv'],
+ format => 'csv')
+```
-#### CSV Format
+#### `S3` Function
-Use the `csv` format to read from CSV. This choice selects the Druid `CsvInputFormat` class.
+The `S3` table function represents the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) which reads
+files from an S3 bucket. The function accepts the following parameters to specify the
+objects to read:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+* `uris` (`ARRAY` of `VARCHAR`)
+* `prefix` (`VARCHAR`) - Corresponds to the JSON `prefixes` property, but allows a single
+ prefix.
+* `bucket` (`VARCHAR`) - Corresponds to the `bucket` field of the `objects` JSON field. SQL
+ does not have syntax for an array of objects. Instead, this function taks a single bucket,
+ and one or more objects within that bucket.
+* `paths` (`ARRAY` of `VARCHAR`) - Corresponds to the `path` fields of the `object` JSON field.
+ All paths are within the single `bucket` parameter.
-MSQ does not have the ability to infer schema from a CSV, file, so the `findColumnsFromHeader` property
-is unavailable. Instead, Columns are given using the `EXTEND` syntax described above.
+The S3 input source accepts one of the following patterns:
-#### Delimited Text Format
+* `uris` - A list of fully-qualified object URIs.
+* `prefixes` - A list of fully-qualified "folder" prefixes.
+* `bucket` and `paths` - A list of objects relative to the given bucket path.
+
+The `S3` function also accepts the following security properties:
-Use the `tsv` format to read from an arbitrary delimited (CSV-like) file such as tab-delimited,
-pipe-delimited, etc. This choice selects the Druid `DelimitedInputFormat` class.
+* `accessKeyId` (`VARCHAR`)
+* `secretAccessKey` (`VARCHAR`)
+* `assumeRoleArn` (`VARCHAR`)
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `delimiter` | The delimiter which separates fields. | `delimiter` | Yes |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+The `S3` table function does not support either the `clientConfig` or `proxyConfig`
+JSON properties.
-As noted above, MSQ cannot infer schema using headers. Use `EXTEND` instead.
+If you need the full power of the S3 input source, then consider the use of the `extern`
+function, which accepts the full S3 input source serializd as JSON. Alternatively,
+create a catalog external table that has the full set of properties, leaving just the
+`uris` or `paths` to be provided at query time.
+
+Examples, each of which correspond to an example on the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) page.
+The examples omit the format and schema; however you must remember to provide those
+in an actual query.
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json']))
+```
+
+```sql
+ TABLE(S3(prefixes => ARRAY['s3://foo/bar/', 's3://bar/foo/']))
+```
+
+```sql
+ -- Not an exact match for the JSON example: the S3 function allows
+ -- only one bucket.
+ TABLE(S3(bucket => 's3://foo`,
+ paths => ARRAY['bar/file1.json', 'foo/file2.json']))
+```
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json'],
+ accessKeyId => 'KLJ78979SDFdS2',
+ secretAccessKey => 'KLS89s98sKJHKJKJH8721lljkd'))
+```
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json'],
+ accessKeyId => 'KLJ78979SDFdS2',
+ secretAccessKey => 'KLS89s98sKJHKJKJH8721lljkd',
+ assumeRoleArn => 'arn:aws:iam::2981002874992:role/role-s3'))
+```
+
+#### Input Format
+
+Each of the table functions above requires that you specify a format using the `format`
+parameter which accepts a value the same as the format names used for `EXTERN` and described
+for [each input source](../ingestion/native-batch-input-sources.md).
+
+#### CSV Format
+
+The `csv` format selects the [CSV input format](../ingestion/data-formats.md#csv).
+Parameters:
+
+* `listDelimiter` (`VARCHAR`)
+* `skipHeaderRows` (`BOOLEAN`)
+
+#### Delimited Text Format
+
+The `tsv` format selects the [TSV (Delimited) input format](../ingestion/data-formats.md#tsv-delimited).
+Parameters:
+
+* `delimiter` (`VARCHAR`)
+* `listDelimiter` (`VARCHAR`)
+* `skipHeaderRows` (`BOOLEAN`)
#### JSON Format
-Use the `json` format to read from a JSON input source. This choice selects the Druid `JsonInputFormat` class.
+The `json` format selects the
+[JSON input format](../ingestion/data-formats.html#json).
+The JSON format accepts no additional parameters.
+
+### Parameters
+
+Starting with the Druid 26.0 release, you can use query parameters with MSQ queries. You may find
+that you periodically ingest a new set of files into Druid. Often, the bulk of the query is identical
+for each ingestion: only the list of files (or URIs or objects) changes. For example, for the `S3`
+input source, you will likely ingest from the same bucket and security setup in
+each query; only the specific objects will change. Consider using a query parameter
+to pass the object names:
+
+```sql
+INSERT INTO ...
+SELECT ...
+FROM TABLE(S3(bucket => 's3://foo`,
+ accessKeyId => ?,
+ paths => ?,
+ format => JSON))
+ (a VARCHAR, b BIGINT, ...)
+```
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `keepNulls` | Whether to keep null values. Defaults to `false`. | `keepNullColumns` | No |
+This same technique can be used with the `uris` or `prefixes` parameters instead.
+Function arguments that take an array parameter require an array function in your JSON request.
+For example:
+
+```json
+{
+ "query" : "INSERT INTO ...
+SELECT ...
+FROM TABLE(S3(bucket => 's3://foo`,
+ accessKeyId => ?,
+ paths => ?,
+ format => JSON))
+ (a VARCHAR, b BIGINT, ...)",
+ "parameters": [
+ { "type": "VARCHAR", "value": "ABCD-EF01"},
+ { "type": "VARCHAR", "value": [
Review Comment:
question: should the type here be `VARCHAR` or `ARRAY`? Just checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066150843
##########
docs/multi-stage-query/reference.md:
##########
@@ -280,6 +481,31 @@ The following ISO 8601 periods are supported for `TIME_FLOOR`:
- P3M
- P1Y
+The string constant can also include any of the keywords mentioned above:
+
+- `HOUR` - Same as `'PT1H'`
+- `DAY` - Same as `'P1D'`
Review Comment:
is `WEEK` an option here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066355611
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
Should this return the enum name as it is or the Period string (ex. this would return `MINUTE`, or `P1M`)? If the user provides the period name on the next go, that seems that it will fail the validation here, as it wont be a valid Period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067627846
##########
server/src/main/java/org/apache/druid/catalog/model/table/LocalInputSourceDefn.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.base.Strings;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.LocalInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Definition for a {@link LocalInputSource}.
+ */
+public class LocalInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = LocalInputSource.TYPE_KEY;
+
+ /**
+ * Base directory for file or filter operations. If not provided,
+ * then the servers current working directory is assumed, which is
+ * typically valid only for sample data.
+ */
+ public static final String BASE_DIR_PARAMETER = "baseDir";
+
+ // Note name "fileFilter", not "filter". These properties mix in with
+ // others and "filter" is a bit too generic in that context.
+ public static final String FILTER_PARAMETER = "filter";
+ public static final String FILES_PARAMETER = "files";
+
+ protected static final String BASE_DIR_FIELD = "baseDir";
+ protected static final String FILES_FIELD = "files";
+ protected static final String FILTER_FIELD = "filter";
+
+ private static final ParameterDefn FILTER_PARAM_DEFN =
+ new Parameter(FILTER_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn FILES_PARAM_DEFN =
+ new Parameter(FILES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+
+ @Override
+ public String typeValue()
+ {
+ return LocalInputSource.TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return LocalInputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasBaseDir = sourceMap.containsKey(BASE_DIR_FIELD);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ final boolean hasFilter = !Strings.isNullOrEmpty(CatalogUtils.getString(sourceMap, FILTER_FIELD));
+
+ if (!hasBaseDir && !hasFiles) {
+ throw new IAE(
+ "A local input source requires one property of %s or %s",
+ BASE_DIR_FIELD,
+ FILES_FIELD
+ );
+ }
+ if (!hasBaseDir && hasFilter) {
+ throw new IAE(
+ "If a local input source sets property %s, it must also set property %s",
+ FILTER_FIELD,
+ BASE_DIR_FIELD
+ );
+ }
+ if (hasBaseDir && hasFiles) {
+ throw new IAE(
+ "A local input source accepts only one of %s or %s",
+ BASE_DIR_FIELD,
+ FILES_FIELD
+ );
+ }
+ super.validate(table);
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return Arrays.asList(
+ new Parameter(BASE_DIR_PARAMETER, ParameterType.VARCHAR, true),
+ FILTER_PARAM_DEFN,
+ FILES_PARAM_DEFN
+ );
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, LocalInputSource.TYPE_KEY);
+
+ final String baseDirParam = CatalogUtils.getString(args, BASE_DIR_PARAMETER);
+ final List<String> filesParam = CatalogUtils.getStringArray(args, FILES_PARAMETER);
+ final String filterParam = CatalogUtils.getString(args, FILTER_PARAMETER);
+ final boolean hasBaseDir = !Strings.isNullOrEmpty(baseDirParam);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(filesParam);
+ final boolean hasFilter = !Strings.isNullOrEmpty(filterParam);
+ if (!hasBaseDir && !hasFiles) {
+ throw new IAE(
+ "A local input source requires one parameter of %s or %s",
+ BASE_DIR_PARAMETER,
+ FILES_PARAMETER
+ );
+ }
+ if (hasBaseDir && hasFiles) {
+ if (hasFilter) {
+ throw new IAE(
+ "A local input source can set parameter %s or %s, but not both.",
+ FILES_PARAMETER,
+ FILTER_PARAMETER
+ );
+ }
+ // Special workaround: if the user provides a base dir, and files, convert
+ // the files to be absolute paths relative to the base dir. Then, remove
+ // the baseDir, since the Druid input source does not allow both a baseDir
+ // and a list of files.
+ jsonMap.put(FILES_FIELD, absolutePath(baseDirParam, filesParam));
+ return;
+ }
+ if (!hasBaseDir && !Strings.isNullOrEmpty(filterParam)) {
+ throw new IAE(
+ "If a local input source sets parameter %s, it must also set parameter %s",
+ FILTER_PARAMETER,
+ BASE_DIR_PARAMETER
+ );
+ }
+ if (hasBaseDir) {
+ jsonMap.put(BASE_DIR_FIELD, baseDirParam);
+ }
+ if (hasFiles) {
+ jsonMap.put(FILES_FIELD, filesParam);
+ }
+ if (filterParam != null) {
+ jsonMap.put(FILTER_FIELD, filterParam);
+ }
+ }
+
+ private List<String> absolutePath(String baseDirPath, List<String> files)
+ {
+ final File baseDir = new File(baseDirPath);
+ return files.stream()
+ .map(f -> new File(baseDir, f).toString())
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public TableFunction partialTableFn(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ final boolean hasFilter = !Strings.isNullOrEmpty(CatalogUtils.getString(sourceMap, FILTER_FIELD));
+ List<ParameterDefn> params = new ArrayList<>();
+ if (!hasFiles && !hasFilter) {
+ params.add(FILES_PARAM_DEFN);
+ params.add(FILTER_PARAM_DEFN);
+ }
+
+ // Does the table define a format?
+ if (table.inputFormatMap == null) {
+ params = addFormatParameters(params);
+ }
+ return new PartialTableFunction(table, params);
+ }
+
+ @Override
+ protected ExternalTableSpec convertCompletedTable(
+ final ResolvedExternalTable table,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns
+ )
+ {
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final boolean hasFiles = !CollectionUtils.isNullOrEmpty(CatalogUtils.safeGet(sourceMap, FILES_FIELD, List.class));
+ if (hasFiles) {
+ if (!args.isEmpty()) {
+ throw new IAE("The local input source has a file list: do not provide other arguments");
+ }
+ } else {
+ final String baseDir = CatalogUtils.getString(sourceMap, BASE_DIR_FIELD);
+ if (Strings.isNullOrEmpty(baseDir)) {
+ throw new IAE(
+ "When an inline external table is used with a table function, %s must be set",
+ BASE_DIR_FIELD
+ );
+ }
+ final boolean hasFilter = !Strings.isNullOrEmpty(CatalogUtils.getString(sourceMap, FILTER_FIELD));
+ final List<String> filesParam = CatalogUtils.getStringArray(args, FILES_PARAMETER);
+ final String filterParam = CatalogUtils.getString(args, FILTER_PARAMETER);
+ final boolean hasFilesParam = !CollectionUtils.isNullOrEmpty(filesParam);
+ final boolean hasFilterParam = !Strings.isNullOrEmpty(filterParam);
+ if (!hasFiles && !hasFilter && !hasFilesParam && !hasFilterParam) {
Review Comment:
Condition `!hasFiles` is always `true` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067588648
##########
server/src/main/java/org/apache/druid/catalog/model/table/InputFormats.java:
##########
@@ -45,59 +42,85 @@
import java.util.stream.Collectors;
/**
- * Definition of the input formats which converts from property
- * lists in table specs to subclasses of {@link InputFormat}.
+ * Catalog definitions for the Druid input formats. Maps from catalog and SQL
+ * functions to the Druid input format implementations.
*/
public class InputFormats
{
- public interface InputFormatDefn
- {
- String name();
- String typeTag();
- List<PropertyDefn<?>> properties();
- void validate(ResolvedTable table);
- InputFormat convert(ResolvedTable table);
- }
-
+ /**
+ * Base class for input format definitions.
+ */
public abstract static class BaseFormatDefn implements InputFormatDefn
{
- private final String name;
- private final String typeTag;
- private final List<PropertyDefn<?>> properties;
-
- public BaseFormatDefn(
- final String name,
- final String typeTag,
- final List<PropertyDefn<?>> properties
- )
+ /**
+ * The set of SQL function parameters available when the format is
+ * specified via a SQL function. The parameters correspond to input format
+ * properties, but typically have simpler names and must require only simple
+ * scalar types of the kind that SQL can provide. Each subclass must perform
+ * conversion to the type required for Jackson conversion.
+ */
+ private final List<ParameterDefn> parameters;
+
+ public BaseFormatDefn(List<ParameterDefn> parameters)
{
- this.name = name;
- this.typeTag = typeTag;
- this.properties = properties;
+ this.parameters = parameters == null ? Collections.emptyList() : parameters;
}
@Override
- public String name()
+ public List<ParameterDefn> parameters()
{
- return name;
+ return parameters;
}
+ /**
+ * The target input format class for Jackson conversions.
+ */
+ protected abstract Class<? extends InputFormat> inputFormatClass();
+
@Override
- public String typeTag()
+ public void validate(ResolvedExternalTable table)
{
- return typeTag;
+ // Bare-bones validation: the format has to convert to the proper object.
+ // Subclasses should replace this with something fancier where needed.
+
+ if (table.inputFormatMap != null) {
+ convertFromTable(table);
+ }
}
- @Override
- public List<PropertyDefn<?>> properties()
+ /**
+ * Convert columns from the {@link ColumnSpec} format used by the catalog to the
+ * list of names form requires by input formats.
+ */
+ protected void convertColumns(Map<String, Object> jsonMap, List<ColumnSpec> columns)
{
- return properties;
+ List<String> cols = columns
+ .stream()
+ .map(col -> col.name())
+ .collect(Collectors.toList());
+ jsonMap.put("columns", cols);
Review Comment:
why are columns part of the inputFormat / why are they being mucked with here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] imply-cheddar commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1068753123
##########
docs/multi-stage-query/reference.md:
##########
@@ -280,6 +481,31 @@ The following ISO 8601 periods are supported for `TIME_FLOOR`:
- P3M
- P1Y
+The string constant can also include any of the keywords mentioned above:
+
+- `HOUR` - Same as `'PT1H'`
+- `DAY` - Same as `'P1D'`
Review Comment:
Please don't make WEEK an option. Kill WEEK, it's not commutable with any other granularity larger than it (MONTH, YEAR, etc.) and generates oddities when used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
maytasm commented on PR #13627:
URL: https://github.com/apache/druid/pull/13627#issuecomment-1386170467
@paul-rogers
Is this PR suppose to be merged?
I don't see any approval and just wanted to make sure that this is not a mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066355611
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
Should this return the enum name as it is or the Period string (ex. this would return `MINUTE`, or `PT1M`)? If the user provides the granularity enum name on the next go, that seems that it will fail the validation here, as it wont be a valid Period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066563594
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefn.java:
##########
@@ -47,35 +46,30 @@ public class TableDefn extends ObjectDefn
*/
public static final String DESCRIPTION_PROPERTY = "description";
- private final Map<String, ColumnDefn> columnDefns;
+ private final Map<String, PropertyDefn<?>> columnProperties;
public TableDefn(
final String name,
final String typeValue,
final List<PropertyDefn<?>> properties,
- final List<ColumnDefn> columnDefns
+ final List<PropertyDefn<?>> columnProperties
)
{
super(
name,
typeValue,
CatalogUtils.concatLists(
Collections.singletonList(
- new ModelProperties.StringPropertyDefn(DESCRIPTION_PROPERTY, null)
+ new ModelProperties.StringPropertyDefn(DESCRIPTION_PROPERTY)
),
properties
)
);
- this.columnDefns = columnDefns == null ? Collections.emptyMap() : toColumnMap(columnDefns);
+ this.columnProperties = toPropertyMap(columnProperties);
}
- public static Map<String, ColumnDefn> toColumnMap(final List<ColumnDefn> colTypes)
+ public void bind(TableDefnRegistry registry)
Review Comment:
I assume this will be defined at some later time? what will this function do, javadoc would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1061149392
##########
server/src/main/java/org/apache/druid/catalog/model/table/InputSourceDefn.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.TableDefnRegistry;
+
+/**
+ * Metadata definition for one Druid input source.
+ *
+ * @see {@link ExternalTableDefn} for a detailed explanation.
+ */
+public interface InputSourceDefn
+{
+ /**
+ * Gather information about the set of format definitions.
+ */
+ void bind(TableDefnRegistry registry);
+
+ /**
+ * Type value for this format: same as the type string used in the serialized
+ * JSON for this input source. Used as the key for this definition within the
+ * table registry, and associates the serialized JSON with the corresponding
+ * input source definition.
+ */
+ String typeValue();
+
+ /**
+ * Given a external table catalog spec, with the JSON input source and format
+ * properties parsed to generic Java maps, validate that the properties are
+ * valid prior to saving the spec into the catalog.
+ *
+ * @param table a catalog table spec with the input source and input format
Review Comment:
## Spurious Javadoc @param tags
@param tag "table" does not match any actual parameter of method "validate()".
[Show more details](https://github.com/apache/druid/security/code-scanning/3589)
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/ExternalOperatorConversion.java:
##########
@@ -19,95 +19,112 @@
package org.apache.druid.sql.calcite.external;
+import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.inject.Inject;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.sql.type.OperandTypes;
-import org.apache.calcite.sql.type.ReturnTypes;
-import org.apache.calcite.sql.type.SqlTypeFamily;
-import org.apache.calcite.sql.validate.SqlUserDefinedTableMacro;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.table.BaseTableFunction;
+import org.apache.druid.catalog.model.table.ExternalTableSpec;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
import org.apache.druid.guice.annotations.Json;
+import org.apache.druid.java.util.common.IAE;
import org.apache.druid.segment.column.RowSignature;
-import org.apache.druid.server.security.Action;
-import org.apache.druid.server.security.Resource;
-import org.apache.druid.server.security.ResourceAction;
-import org.apache.druid.server.security.ResourceType;
-import org.apache.druid.sql.calcite.expression.AuthorizableOperator;
-import org.apache.druid.sql.calcite.expression.DruidExpression;
-import org.apache.druid.sql.calcite.expression.SqlOperatorConversion;
-import org.apache.druid.sql.calcite.planner.DruidTypeSystem;
-import org.apache.druid.sql.calcite.planner.PlannerContext;
-import javax.annotation.Nullable;
-import java.util.Collections;
-import java.util.Set;
-import java.util.stream.Collectors;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
/**
* Registers the "EXTERN" operator, which is used in queries like
- * "INSERT INTO dst SELECT * FROM TABLE(EXTERN(...))".
+ * <pre>{@code
+ * INSERT INTO dst SELECT * FROM TABLE(EXTERN(
+ * "<input source>",
+ * "<input format>",
+ * "<signature>"))
*
- * This class is exercised in CalciteInsertDmlTest but is not currently exposed to end users.
+ * INSERT INTO dst SELECT * FROM TABLE(EXTERN(
+ * inputSource => "<input source>",
+ * inputFormat => "<input format>"))
+ * EXTEND (<columns>)
+ * }</pre>
+ * Where either the by-position or by-name forms are usable with either
+ * a Druid JSON signature, or an SQL {@code EXTEND} list of columns.
+ * As with all table functions, the {@code EXTEND} is optional.
*/
-public class ExternalOperatorConversion implements SqlOperatorConversion
+public class ExternalOperatorConversion extends CatalogExternalTableOperatorConversion
{
public static final String FUNCTION_NAME = "EXTERN";
- // Resource that allows reading external data via SQL.
- public static final ResourceAction EXTERNAL_RESOURCE_ACTION =
- new ResourceAction(new Resource("EXTERNAL", ResourceType.EXTERNAL), Action.READ);
+ public static final String INPUT_SOURCE_PARAM = "inputSource";
+ public static final String INPUT_FORMAT_PARAM = "inputFormat";
+ public static final String SIGNATURE_PARAM = "signature";
- private final SqlUserDefinedTableMacro operator;
-
- @Inject
- public ExternalOperatorConversion(@Json final ObjectMapper jsonMapper)
- {
- this.operator = new ExternalOperator(new ExternalTableMacro(jsonMapper));
- }
-
- @Override
- public SqlOperator calciteOperator()
+ /**
+ * The use of a table function allows the use of optional arguments,
+ * so that the signature can be given either as the original-style
+ * serialized JSON signature, or the updated SQL-style EXTEND clause.
+ */
+ private static class ExternFunction extends BaseTableFunction
{
- return operator;
- }
-
- @Nullable
- @Override
- public DruidExpression toDruidExpression(PlannerContext plannerContext, RowSignature rowSignature, RexNode rexNode)
- {
- return null;
- }
-
- private static class ExternalOperator extends SqlUserDefinedTableMacro implements AuthorizableOperator
- {
- public ExternalOperator(final ExternalTableMacro macro)
+ public ExternFunction()
{
- super(
- new SqlIdentifier(FUNCTION_NAME, SqlParserPos.ZERO),
- ReturnTypes.CURSOR,
- null,
- OperandTypes.sequence(
- macro.signature(),
- OperandTypes.family(SqlTypeFamily.STRING),
- OperandTypes.family(SqlTypeFamily.STRING),
- OperandTypes.family(SqlTypeFamily.STRING)
- ),
- macro.getParameters()
- .stream()
- .map(parameter -> parameter.getType(DruidTypeSystem.TYPE_FACTORY))
- .collect(Collectors.toList()),
- macro
- );
+ super(Arrays.asList(
+ new Parameter(INPUT_SOURCE_PARAM, ParameterType.VARCHAR, true),
+ new Parameter(INPUT_FORMAT_PARAM, ParameterType.VARCHAR, true),
+
+ // Not required: the user can either provide the signature OR
+ // an EXTEND clause. Checked in the implementation.
+ new Parameter(SIGNATURE_PARAM, ParameterType.VARCHAR, false)
+ ));
}
@Override
- public Set<ResourceAction> computeResources(final SqlCall call)
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
{
- return Collections.singleton(EXTERNAL_RESOURCE_ACTION);
+ try {
+ final String sigValue = CatalogUtils.getString(args, SIGNATURE_PARAM);
+ if (sigValue == null && columns == null) {
+ throw new IAE(
+ "EXTERN requires either a %s value or an EXTEND clause",
+ SIGNATURE_PARAM
+ );
+ }
+ if (sigValue != null && columns != null) {
+ throw new IAE(
+ "EXTERN requires either a $s value or an EXTEND clause, but not both",
+ SIGNATURE_PARAM
+ );
Review Comment:
## Unused format argument
This format call refers to 0 argument(s) but supplies 1 argument(s).
[Show more details](https://github.com/apache/druid/security/code-scanning/3595)
##########
server/src/main/java/org/apache/druid/catalog/model/facade/DatasourceFacade.java:
##########
@@ -35,37 +44,179 @@
*/
public class DatasourceFacade extends TableFacade
{
+ private static final Logger LOG = new Logger(DatasourceFacade.class);
+
+ public static class ColumnFacade
+ {
+ public enum Kind
+ {
+ ANY,
+ TIME,
+ DIMENSION,
+ MEASURE
+ }
+
+ private final ColumnSpec spec;
+ private final ParsedType type;
+
+ public ColumnFacade(ColumnSpec spec)
+ {
+ this.spec = spec;
+ if (Columns.isTimeColumn(spec.name()) && spec.sqlType() == null) {
+ // For __time only, force a type if type is null.
+ this.type = TypeParser.TIME_TYPE;
+ } else {
+ this.type = TypeParser.parse(spec.sqlType());
+ }
+ }
+
+ public ColumnSpec spec()
+ {
+ return spec;
+ }
+
+ public ParsedType type()
+ {
+ return type;
+ }
+
+ public boolean hasType()
+ {
+ return type.kind() != ParsedType.Kind.ANY;
+ }
+
+ public boolean isTime()
+ {
+ return type.kind() == ParsedType.Kind.TIME;
+ }
+
+ public boolean isMeasure()
+ {
+ return type.kind() == ParsedType.Kind.MEASURE;
+ }
+
+ public ColumnType druidType()
+ {
+ switch (type.kind()) {
+ case DIMENSION:
+ return Columns.druidType(spec.sqlType());
+ case TIME:
+ return ColumnType.LONG;
+ case MEASURE:
+ return type.measure().storageType;
+ default:
+ return null;
+ }
+ }
+
+ public String sqlStorageType()
+ {
+ if (isTime()) {
+ // Time is special: its storage type is BIGINT, but SQL requires the
+ // type of TIMESTAMP to allow insertion validation.
+ return Columns.TIMESTAMP;
+ } else {
+ return Columns.sqlType(druidType());
+ }
+ }
+
+ @Override
+ public String toString()
+ {
+ return "{spec=" + spec + ", type=" + type + "}";
Review Comment:
## Use of default toString()
Default toString(): ParsedType inherits toString() from Object, and so is not suitable for printing.
[Show more details](https://github.com/apache/druid/security/code-scanning/3597)
##########
server/src/main/java/org/apache/druid/catalog/model/TypeParser.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model;
+
+import org.apache.druid.catalog.model.MeasureTypes.MeasureType;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Parser for the SQL type format used in the catalog.
+ * <ul>
+ * <li>{@code null} - A null type means "use whatever Druid wants."</li>
+ * <li>Scalar - {@code VARCHAR}, {@code BIGINT}, etc.</li>
+ * <li>{@code TIMESTAMP('<grain'>} - truncated timestamp</li>
+ * <li>{@code fn(<type>)} - aggregate measure such as @{code COUNT},
+ * {@code COUNT()} or {@code SUM(BIGINT)}.</li>
+ * </ul>
+ */
+public class TypeParser
+{
+ public static final ParsedType ANY_TYPE = new ParsedType(null, ParsedType.Kind.ANY, null, null);
+ public static final ParsedType TIME_TYPE = new ParsedType(null, ParsedType.Kind.TIME, null, null);
+
+ public static class ParsedType
+ {
+ public enum Kind
+ {
+ ANY,
+ TIME,
+ DIMENSION,
+ MEASURE
+ }
+
+ private final String type;
+ private final Kind kind;
+ private final String timeGrain;
+ private final MeasureType measureType;
+
+ public ParsedType(String type, Kind kind, String timeGrain, MeasureType measureType)
+ {
+ this.type = type;
+ this.kind = kind;
+ this.timeGrain = timeGrain;
+ this.measureType = measureType;
+ }
+
+ public String type()
+ {
+ return type;
+ }
+
+ public Kind kind()
+ {
+ return kind;
+ }
+
+ public String timeGrain()
+ {
+ return timeGrain;
+ }
+
+ public MeasureType measure()
+ {
+ return measureType;
+ }
+ }
+
+ private static class Token
+ {
+ private enum Kind
+ {
+ SYMBOL,
+ OPEN,
+ STRING,
+ COMMA,
+ CLOSE
+ }
+
+ private final Kind kind;
+ private final String value;
+
+ public Token(Kind kind, String value)
+ {
+ this.kind = kind;
+ this.value = value;
+ }
+ }
+
+ private String input;
+ private int posn;
+
+ private TypeParser(String type)
+ {
+ this.input = type;
+ }
+
+ public static ParsedType parse(String type)
+ {
+ if (type == null) {
+ return ANY_TYPE;
+ }
+ return new TypeParser(type).parse();
+ }
+
+ private ParsedType parse()
+ {
+ Token token = parseToken();
+ if (token == null) {
+ return ANY_TYPE;
+ }
+ if (token.kind != Token.Kind.SYMBOL) {
+ throw new IAE("Invalid type name");
+ }
+ final String baseName = StringUtils.toUpperCase(token.value);
+ boolean isTime = Columns.isTimestamp(baseName);
+ token = parseToken();
+ if (token == null) {
+ if (isTime) {
+ return new ParsedType(baseName, ParsedType.Kind.TIME, null, null);
+ } else if (Columns.isScalar(baseName)) {
+ return new ParsedType(baseName, ParsedType.Kind.DIMENSION, null, null);
+ }
+ return analyzeAggregate(baseName, Collections.emptyList());
+ }
+ if (token.kind != Token.Kind.OPEN) {
+ throw new IAE("Invalid type name");
+ }
+ if (!isTime && Columns.isScalar(baseName)) {
+ throw new IAE("Invalid type name");
+ }
+ List<Token> args = new ArrayList<>();
+ token = parseToken();
+ if (token == null) {
+ throw new IAE("Invalid type name");
+ }
+ if (token.kind != Token.Kind.CLOSE) {
+ if (token.kind != Token.Kind.SYMBOL && token.kind != Token.Kind.STRING) {
+ throw new IAE("Invalid type name");
+ }
+ args.add(token);
+ while (true) {
+ token = parseToken();
+ if (token == null) {
+ throw new IAE("Invalid type name");
+ }
+ if (token.kind == Token.Kind.CLOSE) {
+ break;
+ }
+ if (token.kind != Token.Kind.COMMA) {
+ throw new IAE("Invalid type name");
+ }
+ token = parseToken();
+ if (token.kind != Token.Kind.SYMBOL && token.kind != Token.Kind.STRING) {
+ throw new IAE("Invalid type name");
+ }
+ args.add(token);
+ }
+ }
+ token = parseToken();
+ if (token != null) {
+ throw new IAE("Invalid type name");
+ }
+ if (isTime) {
+ return analyzeTimestamp(baseName, args);
+ } else {
+ return analyzeAggregate(baseName, args);
+ }
+ }
+
+ private ParsedType analyzeTimestamp(String baseName, List<Token> args)
Review Comment:
## Useless parameter
The parameter 'baseName' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/3591)
##########
sql/src/main/java/org/apache/druid/sql/SqlPlanningException.java:
##########
@@ -61,22 +60,53 @@
public SqlPlanningException(SqlParseException e)
{
- this(PlanningError.SQL_PARSE_ERROR, e.getMessage());
+ this(e, PlanningError.SQL_PARSE_ERROR, e.getMessage());
}
public SqlPlanningException(ValidationException e)
{
- this(PlanningError.VALIDATION_ERROR, e.getMessage());
+ this(e, PlanningError.VALIDATION_ERROR, e.getMessage());
}
public SqlPlanningException(CalciteContextException e)
{
- this(PlanningError.VALIDATION_ERROR, e.getMessage());
+ this(e, PlanningError.VALIDATION_ERROR, e.getMessage());
}
public SqlPlanningException(PlanningError planningError, String errorMessage)
{
- this(planningError.errorCode, errorMessage, planningError.errorClass);
+ this(
+ null,
+ planningError.errorCode,
+ errorMessage,
+ planningError.errorClass
+ );
+ }
+
+ /**
+ * Constructor which preserves the source exception. Essential during debugging
+ * to see what really failed. If we don't want to show the user the underlying
+ * error, then don't serialize it out to the user. We must still preserve the
+ * cause internally to save time and sanity for developers.
+ */
+ public SqlPlanningException(Exception e, PlanningError planningError, String errorMessage)
+ {
+ this(
+ e,
+ planningError.errorCode,
+ errorMessage,
+ planningError.errorClass
+ );
+ }
+
+ private SqlPlanningException(
+ Exception e,
Review Comment:
## Useless parameter
The parameter 'e' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/3592)
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ private Map<String, Object> inputSourceMap;
+ private Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ if (this.inputSourceMap == null || this.inputSourceMap.isEmpty()) {
Review Comment:
## Useless null check
This check is useless. [this.inputSourceMap](1) cannot be null at this check, since [new HashMap<String,Object>(...)](2) always is non-null.
[Show more details](https://github.com/apache/druid/security/code-scanning/3593)
##########
server/src/test/java/org/apache/druid/catalog/model/table/InlineInputSourceDefnTest.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableMetadata;
+import org.apache.druid.catalog.model.table.InputFormats.CsvFormatDefn;
+import org.apache.druid.catalog.model.table.InputFormats.FlatTextFormatDefn;
+import org.apache.druid.data.input.impl.CsvInputFormat;
+import org.apache.druid.data.input.impl.InlineInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.junit.Test;
+
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class InlineInputSourceDefnTest extends BaseExternTableTest
+{
+ @Test
+ public void testValidateEmptyInputSource()
+ {
+ // No data property: not valid
+ TableMetadata table = TableBuilder.external("foo")
+ .inputSource(ImmutableMap.of("type", InlineInputSource.TYPE_KEY))
+ .inputFormat(CSV_FORMAT)
+ .column("x", Columns.VARCHAR)
+ .build();
+ ResolvedTable resolved = registry.resolve(table.spec());
+ assertThrows(IAE.class, () -> resolved.validate());
+ }
+
+ @Test
+ public void testValidateNoFormat()
+ {
+ // No format: not valid. For inline, format must be provided to match data
+ TableMetadata table = TableBuilder.external("foo")
+ .inputSource(toMap(new InlineInputSource("a\n")))
+ .column("x", Columns.VARCHAR)
+ .build();
+ ResolvedTable resolved = registry.resolve(table.spec());
+ assertThrows(IAE.class, () -> resolved.validate());
+ }
+
+ @Test
+ public void testValidateNoColumns() throws URISyntaxException
+ {
+ TableMetadata table = TableBuilder.external("foo")
+ .inputSource(toMap(new InlineInputSource("a\n")))
+ .inputFormat(CSV_FORMAT)
+ .build();
+ ResolvedTable resolved = registry.resolve(table.spec());
+ assertThrows(IAE.class, () -> resolved.validate());
+ }
+
+ @Test
+ public void testValidateGood()
+ {
+ TableMetadata table = TableBuilder.external("foo")
+ .inputSource(toMap(new InlineInputSource("a\n")))
+ .inputFormat(CSV_FORMAT)
+ .column("x", Columns.VARCHAR)
+ .build();
+ ResolvedTable resolved = registry.resolve(table.spec());
+ resolved.validate();
+ }
+
+ @Test
+ public void testFullTableFnBasics()
+ {
+ InputSourceDefn defn = registry.inputSourceDefnFor(InlineInputSourceDefn.TYPE_KEY);
+ TableFunction fn = defn.adHocTableFn();
+ assertNotNull(fn);
+ assertTrue(hasParam(fn, InlineInputSourceDefn.DATA_PROPERTY));
+ assertTrue(hasParam(fn, FormattedInputSourceDefn.FORMAT_PARAMETER));
+ assertTrue(hasParam(fn, FlatTextFormatDefn.LIST_DELIMITER_PARAMETER));
+ }
+
+ @Test
+ public void testMissingArgs()
+ {
+ InputSourceDefn defn = registry.inputSourceDefnFor(InlineInputSourceDefn.TYPE_KEY);
+ TableFunction fn = defn.adHocTableFn();
+ assertThrows(IAE.class, () -> fn.apply("x", new HashMap<>(), Collections.emptyList(), mapper));
+ }
+
+ @Test
+ public void testMissingFormat()
+ {
+ InputSourceDefn defn = registry.inputSourceDefnFor(InlineInputSourceDefn.TYPE_KEY);
+ TableFunction fn = defn.adHocTableFn();
+ Map<String, Object> args = new HashMap<>();
Review Comment:
## Container contents are never accessed
The contents of this container are never accessed.
[Show more details](https://github.com/apache/druid/security/code-scanning/3594)
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ private Map<String, Object> inputSourceMap;
+ private Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ if (this.inputSourceMap == null || this.inputSourceMap.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ map = table.mapProperty(ExternalTableDefn.FORMAT_PROPERTY);
+ this.inputFormatMap = map == null ? null : new HashMap<>(map);
+ }
+
+ public ResolvedTable resolvedTable()
+ {
+ return table;
+ }
+
+ public Map<String, Object> formatMap()
+ {
+ return inputFormatMap;
+ }
+
+ public Map<String, Object> sourceMap()
Review Comment:
## Exposing internal representation
sourceMap exposes the internal representation stored in field inputSourceMap. The value may be modified [after this call to sourceMap](1).
sourceMap exposes the internal representation stored in field inputSourceMap. The value may be modified [after this call to sourceMap](2).
[Show more details](https://github.com/apache/druid/security/code-scanning/3596)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066564475
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefn.java:
##########
@@ -99,26 +93,13 @@ public void validateColumns(List<ColumnSpec> columns, ObjectMapper jsonMapper)
if (!names.add(colSpec.name())) {
throw new IAE("Duplicate column name: " + colSpec.name());
}
- ColumnDefn.ResolvedColumn resolvedCol = resolveColumn(colSpec);
- resolvedCol.validate(jsonMapper);
+ colSpec.validate();
+ validateColumn(colSpec);
}
}
- /**
- * Resolve the column type to produce a composite object that holds
- * both the definition and the column spec.
- */
- public ColumnDefn.ResolvedColumn resolveColumn(ColumnSpec spec)
+ protected void validateColumn(ColumnSpec colSpec)
Review Comment:
I assume this will be defined at some later time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066568661
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefnRegistry.java:
##########
@@ -55,40 +61,76 @@
{
// Temporary list of Druid-define table definitions. This should come from
// Guice later to allow extensions to define table types.
- private static final TableDefn[] TABLE_DEFNS = {
+ private static final List<TableDefn> BUILTIN_TABLE_DEFNS = Arrays.asList(
new DatasourceDefn(),
- new InlineTableDefn(),
- new HttpTableDefn(),
- new LocalTableDefn()
- };
+ new ExternalTableDefn()
+ );
+ private static final List<InputSourceDefn> BUILTIN_INPUT_SOURCE_DEFNS = Arrays.asList(
+ new InlineInputSourceDefn(),
+ new HttpInputSourceDefn(),
+ new LocalInputSourceDefn()
+ );
+ private static final List<InputFormatDefn> BUILTIN_INPUT_FORMAT_DEFNS = Arrays.asList(
+ new InputFormats.CsvFormatDefn(),
+ new InputFormats.DelimitedFormatDefn(),
+ new InputFormats.JsonFormatDefn()
+ );
- private final Map<String, TableDefn> defns;
+ private final Map<String, TableDefn> tableDefns;
+ private final Map<String, InputSourceDefn> inputSourceDefns;
+ private final Map<String, InputFormatDefn> inputFormatDefns;
private final ObjectMapper jsonMapper;
public TableDefnRegistry(
- final TableDefn[] defns,
+ @Nullable final List<TableDefn> tableDefnExtns,
+ @Nullable final List<InputSourceDefn> inputSourceDefnExtns,
+ @Nullable final List<InputFormatDefn> inputFormatDefnExtns,
final ObjectMapper jsonMapper
)
{
- ImmutableMap.Builder<String, TableDefn> builder = ImmutableMap.builder();
- for (TableDefn defn : defns) {
- builder.put(defn.typeValue(), defn);
- }
- this.defns = builder.build();
this.jsonMapper = jsonMapper;
+ final List<TableDefn> tableDefns = CatalogUtils.concatLists(tableDefnExtns, BUILTIN_TABLE_DEFNS);
+ final List<InputSourceDefn> inputSourceDefns = CatalogUtils.concatLists(inputSourceDefnExtns, BUILTIN_INPUT_SOURCE_DEFNS);
+ final List<InputFormatDefn> inputFormatDefns = CatalogUtils.concatLists(inputFormatDefnExtns, BUILTIN_INPUT_FORMAT_DEFNS);
+
+ ImmutableMap.Builder<String, TableDefn> tableBuilder = ImmutableMap.builder();
+ for (TableDefn defn : tableDefns) {
+ tableBuilder.put(defn.typeValue(), defn);
+ }
+ this.tableDefns = tableBuilder.build();
+
+ ImmutableMap.Builder<String, InputSourceDefn> sourceBuilder = ImmutableMap.builder();
+ for (InputSourceDefn defn : inputSourceDefns) {
+ sourceBuilder.put(defn.typeValue(), defn);
+ }
+ this.inputSourceDefns = sourceBuilder.build();
+
+ ImmutableMap.Builder<String, InputFormatDefn> formatBuilder = ImmutableMap.builder();
+ for (InputFormatDefn defn : inputFormatDefns) {
+ formatBuilder.put(defn.typeValue(), defn);
+ }
+ this.inputFormatDefns = formatBuilder.build();
+
+ // Initialize all items once the entire set of bindings is defined.
+ for (InputSourceDefn defn : inputSourceDefns) {
+ defn.bind(this);
+ }
+ for (TableDefn defn : tableDefns) {
+ defn.bind(this);
+ }
Review Comment:
should we bind the inputFormatDefns too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066569420
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefnRegistry.java:
##########
@@ -102,10 +144,25 @@ public ResolvedTable resolve(TableSpec spec)
if (Strings.isNullOrEmpty(type)) {
throw new IAE("The table type is required.");
}
- TableDefn defn = defns.get(type);
+ TableDefn defn = tableDefns.get(type);
if (defn == null) {
throw new IAE("Table type [%s] is not valid.", type);
}
return new ResolvedTable(defn, spec, jsonMapper);
}
+
+ public InputSourceDefn inputSourceDefnFor(String type)
+ {
+ return type == null ? null : inputSourceDefns.get(type);
+ }
+
+ public InputFormatDefn inputFormatDefnFor(String type)
+ {
+ return type == null ? null : inputFormatDefns.get(type);
Review Comment:
nit: I dont think you need the null check do you? If null, should just return null? Or why is this check not being done for `formats()`?
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefnRegistry.java:
##########
@@ -102,10 +144,25 @@ public ResolvedTable resolve(TableSpec spec)
if (Strings.isNullOrEmpty(type)) {
throw new IAE("The table type is required.");
}
- TableDefn defn = defns.get(type);
+ TableDefn defn = tableDefns.get(type);
if (defn == null) {
throw new IAE("Table type [%s] is not valid.", type);
}
return new ResolvedTable(defn, spec, jsonMapper);
}
+
+ public InputSourceDefn inputSourceDefnFor(String type)
+ {
+ return type == null ? null : inputSourceDefns.get(type);
Review Comment:
nit: I dont think you need the null check do you? If null, should just return null? Or why is this check not being done for formats()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067568739
##########
server/src/main/java/org/apache/druid/catalog/model/table/HttpInputSourceDefn.java:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.HttpInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.metadata.DefaultPasswordProvider;
+import org.apache.druid.metadata.EnvironmentVariablePasswordProvider;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Definition of an HTTP input source source.
+ * <p>
+ * Provides a parameterized form where the user defines a value for the
+ * {@code uriTemplate} table property in the table spec, then provides the partial URLs
+ * in a table function to use for that one query. The final URIs are created by combining
+ * the template and the arguments. Example:
+ * <li>{@code uriTemplate} property): "http://example.com/data/kttm-{}.json"</li>
+ * <li>{@code uris} function argument: "22-Nov-21, 22-Nov-22"</li>
+ * </ul>
+ * <p>
+ * When the template is used, the format is optional: it can be provided either with
+ * the table spec or at runtime, depending on what the user wants to accomplish. In the
+ * above, where the ".json" is encoded in the template, it makes sense to include the format
+ * with the spec. If the template was "http://example.com/data/{}", and the data comes in
+ * multiple formats, it might make sense to specify the format in the query. In this case,
+ * the table spec acts more like a connection.
+ * <p>
+ * If the template is not used, then the {@code uris} property must be provided in the
+ * table spec, along with the corresponding format.
+ * <p>
+ * The above semantics make a bit more sense when we realize that the spec can also
+ * provide a user name and password. When those are provided, then the input source must
+ * name a single site: the one for which the credentials are valid. Given this, the only
+ * table spec that makes sense is one where the URI is defined: either as a template or
+ * explicitly.
+ * <p>
+ * When used as an ad-hoc function, the user specifies the uris and optional user name
+ * and password: the template is not available (or useful) in the ad-hoc case.
+ * <p>
+ * Table function parameters are cleaned up relative to the input source field names to
+ * make them a bit easier to use.
+ */
+public class HttpInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = HttpInputSource.TYPE_KEY;
+
+ // Catalog properties that map to fields in the HttpInputSource. See
+ // that class for the meaning of these properties.
+
+ public static final String URI_TEMPLATE_PROPERTY = "uriTemplate";
+
+ public static final String URIS_PARAMETER = "uris";
+
+ // Note, cannot be the simpler "user" since USER is a reserved word in SQL
+ // and we don't want to require users to quote "user" each time it is used.
+ public static final String USER_PARAMETER = "userName";
+ public static final String PASSWORD_PARAMETER = "password";
+ public static final String PASSWORD_ENV_VAR_PARAMETER = "passwordEnvVar";
+
+ private static final List<ParameterDefn> URI_PARAMS = Arrays.asList(
+ new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true)
+ );
+
+ private static final List<ParameterDefn> USER_PWD_PARAMS = Arrays.asList(
+ new Parameter(USER_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(PASSWORD_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(PASSWORD_ENV_VAR_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the HttpInputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PASSWORD_FIELD = "httpAuthenticationPassword";
+ private static final String USERNAME_FIELD = "httpAuthenticationUsername";
+
+ @Override
+ public String typeValue()
+ {
+ return TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return HttpInputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUri = sourceMap.containsKey(URIS_FIELD);
+ final String uriTemplate = table.resolvedTable().stringProperty(URI_TEMPLATE_PROPERTY);
+ final boolean hasTemplate = uriTemplate != null;
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (!hasUri && !hasTemplate) {
+ throw new IAE(
+ "External HTTP tables must provide either a URI or a %s property",
+ URI_TEMPLATE_PROPERTY
+ );
+ }
+ if (hasUri && hasTemplate) {
+ throw new IAE(
+ "External HTTP tables must provide only one of a URI or a %s property",
+ URI_TEMPLATE_PROPERTY
+ );
+ }
+ if (hasUri && !hasFormat) {
+ throw new IAE(
+ "An external HTTP table with a URI must also provide the corresponding format"
+ );
+ }
+ if (hasUri && !hasColumns) {
+ throw new IAE(
+ "An external HTTP table with a URI must also provide the corresponding columns"
+ );
+ }
+ if (hasTemplate) {
+
+ // Verify the template
+ templateMatcher(uriTemplate);
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ try {
+ sourceMap.put(
+ URIS_FIELD,
+ Collections.singletonList(new URI("https://bogus.com/file"))
+ );
+ }
+ catch (Exception e) {
+ throw new ISE(e, "URI parse failed");
+ }
+ }
+ super.validate(table);
+ }
+
+ private Matcher templateMatcher(String uriTemplate)
+ {
+ Pattern p = Pattern.compile("\\{}");
+ Matcher m = p.matcher(uriTemplate);
+ if (!m.find()) {
+ throw new IAE(
+ "Value [%s] for property %s must include a '{}' placeholder",
+ uriTemplate,
+ URI_TEMPLATE_PROPERTY
+ );
+ }
+ return m;
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return CatalogUtils.concatLists(URI_PARAMS, USER_PWD_PARAMS);
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, HttpInputSource.TYPE_KEY);
+ convertUriArg(jsonMap, args);
+ convertUserPasswordArgs(jsonMap, args);
+ }
+
+ @Override
+ public TableFunction partialTableFn(ResolvedExternalTable table)
+ {
+ List<ParameterDefn> params = Collections.emptyList();
+
+ // Does the table define URIs?
+ Map<String, Object> sourceMap = table.inputSourceMap;
+ if (!sourceMap.containsKey(URIS_FIELD)) {
+ params = CatalogUtils.concatLists(params, URI_PARAMS);
+ }
+
+ // Does the table define a user or password?
+ if (!sourceMap.containsKey(USERNAME_FIELD) && !sourceMap.containsKey(PASSWORD_FIELD)) {
+ params = CatalogUtils.concatLists(params, USER_PWD_PARAMS);
+ }
+
+ // Does the table define a format?
+ if (table.inputFormatMap == null) {
+ params = addFormatParameters(params);
+ }
+ return new PartialTableFunction(table, params);
+ }
+
+ @Override
+ protected ExternalTableSpec convertCompletedTable(
+ final ResolvedExternalTable table,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns
+ )
+ {
+ // Get URIs from table if defined, else from arguments.
+ final Map<String, Object> sourceMap = new HashMap<>(table.inputSourceMap);
+ final String uriTemplate = table.resolvedTable().stringProperty(URI_TEMPLATE_PROPERTY);
+ if (uriTemplate != null) {
+ convertUriTemplateArgs(sourceMap, uriTemplate, args);
+ } else if (!sourceMap.containsKey(URIS_FIELD)) {
+ convertUriArg(sourceMap, args);
+ }
+
+ // Get user and password from the table if defined, else from arguments.
+ if (!sourceMap.containsKey(USERNAME_FIELD) && !sourceMap.containsKey(PASSWORD_FIELD)) {
+ convertUserPasswordArgs(sourceMap, args);
+ }
+ return convertPartialFormattedTable(table, args, columns, sourceMap);
+ }
+
+ private void convertUriTemplateArgs(Map<String, Object> jsonMap, String uriTemplate, Map<String, Object> args)
+ {
+ List<String> uriStrings = CatalogUtils.getStringArray(args, URIS_PARAMETER);
Review Comment:
Have we verified before this that the `URIS_PARAMETER` is present in the args, or can this lead to NPE somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067588648
##########
server/src/main/java/org/apache/druid/catalog/model/table/InputFormats.java:
##########
@@ -45,59 +42,85 @@
import java.util.stream.Collectors;
/**
- * Definition of the input formats which converts from property
- * lists in table specs to subclasses of {@link InputFormat}.
+ * Catalog definitions for the Druid input formats. Maps from catalog and SQL
+ * functions to the Druid input format implementations.
*/
public class InputFormats
{
- public interface InputFormatDefn
- {
- String name();
- String typeTag();
- List<PropertyDefn<?>> properties();
- void validate(ResolvedTable table);
- InputFormat convert(ResolvedTable table);
- }
-
+ /**
+ * Base class for input format definitions.
+ */
public abstract static class BaseFormatDefn implements InputFormatDefn
{
- private final String name;
- private final String typeTag;
- private final List<PropertyDefn<?>> properties;
-
- public BaseFormatDefn(
- final String name,
- final String typeTag,
- final List<PropertyDefn<?>> properties
- )
+ /**
+ * The set of SQL function parameters available when the format is
+ * specified via a SQL function. The parameters correspond to input format
+ * properties, but typically have simpler names and must require only simple
+ * scalar types of the kind that SQL can provide. Each subclass must perform
+ * conversion to the type required for Jackson conversion.
+ */
+ private final List<ParameterDefn> parameters;
+
+ public BaseFormatDefn(List<ParameterDefn> parameters)
{
- this.name = name;
- this.typeTag = typeTag;
- this.properties = properties;
+ this.parameters = parameters == null ? Collections.emptyList() : parameters;
}
@Override
- public String name()
+ public List<ParameterDefn> parameters()
{
- return name;
+ return parameters;
}
+ /**
+ * The target input format class for Jackson conversions.
+ */
+ protected abstract Class<? extends InputFormat> inputFormatClass();
+
@Override
- public String typeTag()
+ public void validate(ResolvedExternalTable table)
{
- return typeTag;
+ // Bare-bones validation: the format has to convert to the proper object.
+ // Subclasses should replace this with something fancier where needed.
+
+ if (table.inputFormatMap != null) {
+ convertFromTable(table);
+ }
}
- @Override
- public List<PropertyDefn<?>> properties()
+ /**
+ * Convert columns from the {@link ColumnSpec} format used by the catalog to the
+ * list of names form requires by input formats.
+ */
+ protected void convertColumns(Map<String, Object> jsonMap, List<ColumnSpec> columns)
{
- return properties;
+ List<String> cols = columns
+ .stream()
+ .map(col -> col.name())
+ .collect(Collectors.toList());
+ jsonMap.put("columns", cols);
Review Comment:
why are columns part of the inputFormat?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067638663
##########
server/src/main/java/org/apache/druid/catalog/model/table/ResolvedExternalTable.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ResolvedTable;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Internal class to hold the intermediate form of an external table: the
+ * input source and input format properties converted to Java maps, and the
+ * types of each resolved to the corresponding definitions. Used to validate
+ * a table specification, and to convert a table specification to an
+ * {@link ExternalTableSpec} when used in SQL.
+ */
+public class ResolvedExternalTable
+{
+ private final ResolvedTable table;
+ protected final Map<String, Object> inputSourceMap;
+ protected final Map<String, Object> inputFormatMap;
+ private InputSourceDefn inputSourceDefn;
+ private InputFormatDefn inputFormatDefn;
+
+ /**
+ * Construct a resolved external table by extracting the input source
+ * and input format properties, and converting each to a Java map.
+ * Validates that the input source is present: the format is optional.
+ * <p>
+ * Note: does <i>not</i> resolve the input source and input format
+ * definitions: that is done as a separate step when needed.
+ *
+ * @see {@link #resolve(TableDefnRegistry)}.
+ */
+ public ResolvedExternalTable(final ResolvedTable table)
+ {
+ this.table = table;
+ Map<String, Object> map = table.mapProperty(ExternalTableDefn.SOURCE_PROPERTY);
+ if (map == null || map.isEmpty()) {
+ throw new IAE("%s property is required", ExternalTableDefn.SOURCE_PROPERTY);
+ }
+ this.inputSourceMap = new HashMap<>(map);
+ map = table.mapProperty(ExternalTableDefn.FORMAT_PROPERTY);
+ this.inputFormatMap = map == null ? null : new HashMap<>(map);
+ }
+
+ public ResolvedTable resolvedTable()
+ {
+ return table;
+ }
+
+ /**
+ * Look up the input source type and input format type to find the corresponding
+ * definitions in the table registry. Throws an exception if the types are not
+ * defined. The input source is required, the format is optional.
+ * <p>
+ * Note, for resolution to work, the name of each definition must be the same as
+ * that used as the type key in the serialized JSON.
+ */
+ public ResolvedExternalTable resolve(TableDefnRegistry registry)
+ {
+ String inputSourceType = CatalogUtils.getString(inputSourceMap, InputSource.TYPE_PROPERTY);
+ if (inputSourceType == null) {
+ throw new IAE("Input source type %s is required", InputSource.TYPE_PROPERTY);
+ }
+ inputSourceDefn = registry.inputSourceDefnFor(inputSourceType);
+ if (inputSourceDefn == null) {
+ throw new IAE("Input source type %s is not registered", inputSourceType);
+ }
+ if (inputFormatMap != null) {
+ String inputFormatType = CatalogUtils.getString(inputFormatMap, InputFormat.TYPE_PROPERTY);
+ if (inputFormatType != null) {
Review Comment:
if the `inputFormatMap` is not null, but does not have the format type set, should that be an error? Right now, it doesn't seem to be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
abhishekrb19 commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067732322
##########
server/src/main/java/org/apache/druid/catalog/model/table/InlineInputSourceDefn.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.InlineInputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Describes an inline input source: one where the data is provided in the
+ * table spec as a series of text lines. Since the data is provided, the input
+ * format is required in the table spec: it cannot be provided at ingest time.
+ * Primarily for testing.
+ */
+public class InlineInputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String TYPE_KEY = InlineInputSource.TYPE_KEY;
+ public static final String DATA_PROPERTY = "data";
+
+ @Override
+ public String typeValue()
+ {
+ return TYPE_KEY;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return InlineInputSource.class;
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return Collections.singletonList(
+ new Parameter(DATA_PROPERTY, ParameterType.VARCHAR_ARRAY, false)
+ );
+ }
+
+ @Override
+ public TableFunction partialTableFn(ResolvedExternalTable table)
+ {
+ return new PartialTableFunction(table, Collections.emptyList());
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ // For inline, format is required to match the data
+ if (table.inputFormatMap == null) {
+ throw new IAE("An inline input source must provide a format.");
+ }
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ throw new IAE("An inline input source must provide one or more columns");
+ }
+
+ super.validate(table);
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, InlineInputSource.TYPE_KEY);
+ List<String> data = CatalogUtils.getStringArray(args, DATA_PROPERTY);
+
+ // Would be nice, from a completeness perspective, for the inline data
+ // source to allow zero rows of data. However, such is not the case.
+ if (CollectionUtils.isNullOrEmpty(data)) {
+ throw new IAE(
+ "An inline table requires one or more rows of data in the '%s' property",
+ DATA_PROPERTY
+ );
+ }
+ jsonMap.put("data", String.join("\n", data));
+ }
+
+ @Override
+ protected ExternalTableSpec convertCompletedTable(
+ final ResolvedExternalTable table,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns
+ )
+ {
+ if (!args.isEmpty()) {
+ throw new ISE("Cannot provide arguments for an inline table");
+ }
+ if (!CollectionUtils.isNullOrEmpty(columns)) {
+ throw new IAE("Cannot provide columns for an inline table");
+ }
+ return convertTable(table);
+ }
+
+ @Override
+ protected void auditInputSource(Map<String, Object> jsonMap)
+ {
+ // Special handling of the data property which, in SQL, is a null-delimited
+ // list of rows. The user will usually provide a trailing newline which should
+ // not be interpreted as an empty data row. That is, if the data ends with
+ // a newline, the inline input source will interpret that as a blank line, oddly.
+ String data = CatalogUtils.getString(jsonMap, "data");
Review Comment:
Replace "data" with `DATA_PROPERTY`:
```suggestion
String data = CatalogUtils.getString(jsonMap, DATA_PROPERTY);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067650355
##########
sql/src/main/java/org/apache/druid/sql/calcite/external/Externals.java:
##########
@@ -145,49 +174,41 @@ public Consistency getConsistency()
}
/**
- * Convert the actual arguments to SQL external table function into a catalog
- * resolved table, then convert that to an external table spec usable by MSQ.
- *
- * @param tableDefn catalog definition of the kind of external table
- * @param parameters the parameters to the SQL table macro
- * @param arguments the arguments that match the parameters. Optional arguments
- * may be null
- * @param schema the external table schema provided by the EXTEND clause
- * @param jsonMapper the JSON mapper to use for value conversions
- * @return a spec with the three values that MSQ needs to create an external table
+ * Convert the list of Calcite function arguments to a map of non-null arguments.
+ * The resulting map must be mutable as processing may rewrite values.
*/
- public static ExternalTableSpec convertArguments(
- final ExternalTableDefn tableDefn,
- final List<FunctionParameter> parameters,
- final List<Object> arguments,
- final SqlNodeList schema,
- final ObjectMapper jsonMapper
+ public static Map<String, Object> convertArguments(
+ final TableFunction fn,
+ final List<Object> arguments
)
{
- final TableBuilder builder = TableBuilder.of(tableDefn);
- for (int i = 0; i < parameters.size(); i++) {
- String name = parameters.get(i).getName();
- Object value = arguments.get(i);
- if (value == null) {
- continue;
+ final List<TableFunction.ParameterDefn> params = fn.parameters();
+ final Map<String, Object> argMap = new HashMap<>();
+ for (int i = 0; i < arguments.size(); i++) {
+ final Object value = arguments.get(i);
+ if (value != null) {
+ argMap.put(params.get(i).name(), value);
}
- PropertyDefn<?> prop = tableDefn.property(name);
- builder.property(name, prop.decodeSqlValue(value, jsonMapper));
}
+ return argMap;
+ }
- // Converts from a list of (identifier, type, ...) pairs to
- // a Druid row signature. The schema itself comes from the
- // Druid-specific EXTEND syntax added to the parser.
+ /**
+ * Converts from a list of (identifier, type, ...) pairs to
+ * list of column specs. The schema itself comes from the
+ * Druid-specific EXTEND syntax added to the parser.
+ */
+ public static List<ColumnSpec> convertColumns(SqlNodeList schema)
+ {
+ final List<ColumnSpec> columns = new ArrayList<>();
for (int i = 0; i < schema.size(); i += 2) {
final String name = convertName((SqlIdentifier) schema.get(i));
- String sqlType = convertType(name, (SqlDataTypeSpec) schema.get(i + 1));
- builder.column(name, sqlType);
+ final String sqlType = convertType(name, (SqlDataTypeSpec) schema.get(i + 1));
+ columns.add(new ColumnSpec(name, sqlType, null));
Review Comment:
since the type can be null now, what does this mean for iterating over the nodeList, is the user expected to put some placeholder identifier to indicate that the type is null / not specified, and if not, will iterating every 2 nodes as were doing here be problematic in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067651827
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidCalciteSchemaModule.java:
##########
@@ -64,7 +64,7 @@ public void configure(Binder binder)
}
@Provides
- @Singleton
+ @LazySingleton
Review Comment:
whats the reason for this change? Worthy of comment here you think, or nah?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066143382
##########
docs/multi-stage-query/reference.md:
##########
@@ -88,93 +118,260 @@ FROM TABLE(
http(
userName => 'bob',
password => 'secret',
- uris => 'http:foo.com/bar.csv',
+ uris => ARRAY['http:example.com/foo.csv', 'http:example.com/bar.csv'],
format => 'csv'
)
) (x VARCHAR, y VARCHAR, z BIGINT)
```
+#### Function Arguments
-The set of table functions and formats is preliminary in this release.
+These table functions are intended for use with the SQL by-name argument syntax
+as shown above. Because the functions include all parameters for all formats,
+using positional calls is both cumbersome and error-prone.
+
+Function argument names are generally the same as the JSON field names, except
+as noted below. Each argument has a SQL type which matches the JSON type. For
+arguments that take a string list in JSON, use the SQL `ARRAY[...]` syntax in
+SQL as shown in the above example.
+
+Array parameters are good candidates for use in parameterized queries. That is:
+
+```sql
+SELECT
+ <column>
+FROM TABLE(
+ http(
+ userName => 'bob',
+ password => 'secret',
+ uris => ?,
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR, z BIGINT)
+```
-#### `HTTP`
+Provide the list of URIs (in this case) as a query parameter in each ingest. Doing
+so is simpler than writing a script to insert the array into the SQL text.
-The `HTTP` table function represents the `HttpInputSource` class in Druid which allows you to
-read from an HTTP server. The function accepts the following arguments:
+#### `HTTP` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `userName` | Basic authentication user name | `httpAuthenticationUsername` | No |
-| `password` | Basic authentication password | `httpAuthenticationPassword` | No |
-| `passwordEnvVar` | Environment variable that contains the basic authentication password| `httpAuthenticationPassword` | No |
-| `uris` | Comma-separated list of URIs to read. | `uris` | Yes |
+The `HTTP` table function represents the
+[HTTP input source](../ingestion/native-batch-input-sources.md#http-input-source)
+to read from an HTTP server. The function accepts the following arguments:
-#### `INLINE`
+* `userName` (`VARCHAR`) - Same as JSON `httpAuthenticationUsername`.
+* `password` (`VARCHAR`) - Same as`httpAuthenticationPassword` when used with the default option.
+* `passwordEnvVar` (`VARCHAR`) - Same as the HTTP `httpAuthenticationPassword` when used with
+ the `"type": "environment"` option.
+* `uris` (`ARRAY` of `VARCHAR`)
-The `INLINE` table function represents the `InlineInputSource` class in Druid which provides
-data directly in the table function. The function accepts the following arguments:
+#### `INLINE` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `data` | Text lines of inline data. Separate lines with a newline. | `data` | Yes |
+The `INLINE` table function represents the
+[Inline input source](../ingestion/native-batch-input-sources.md#inline-input-source)
+which provides data directly in the table function. Parameter:
-#### `LOCALFILES`
+* `data` (`ARRAY` of `VARCHAR`) - Data lines, without a trailing newline, as an array.
-The `LOCALFILES` table function represents the `LocalInputSource` class in Druid which reads
+Example:
+
+```sql
+FROM TABLE(
+ inline(
+ data => ARRAY[
+ "a,b",
+ "c,d"],
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR)
+```
+
+
+#### `LOCALFILES` Function
+
+The `LOCALFILES` table function represents the
+[Local input source](../ingestion/native-batch-input-sources.md#local-input-source) which reads
files from the file system of the node running Druid. This is most useful for single-node
-installations. The function accepts the following arguments:
+installations. The function accepts the following parameters:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `baseDir` | Directory to read from. | `baseDir` | No |
-| `filter` | Filter pattern to read. Example: `*.csv`. | `filter` | No |
-| `files` | Comma-separated list of files to read. | `files` | No |
+* `baseDir`
+* `filter`
+* `files`
-You must either provide the `baseDir` or the list of `files`. You can provide both, in which case
-the files are assumed relative to the `baseDir`. If you provide a `filter`, you must provide the
-`baseDir`.
+When the local files input source is used directly in an `extern` function, or ingestion spec, you
+can provide either `baseDir` and `filter` or `files` but not both. This function, however, allows
+you to provide any of the following combinations:
-#### Table Function Format
+* `baseDir` - Matches all files in the given directory. (Assumes the filter is `*`.)
+* `baseDir` and `filter` - Match files in the given directory using the filter.
+* `baseDir` and `files` - A set of files relative to `baseDir`.
+* `files` - The files should be absolute paths, else they will be computed relative to Druid's
+ working directory (usually the Druid install directory.)
-Each of the table functions above requires that you specify a format.
+Examples:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `format` | The input format, using the same names as for `EXTERN`. | `inputFormat.type` | Yes |
+```sql
+ -- All files in /tmp, which must be CSV files
+ localfiles(baseDir => '/tmp',
+ format => 'csv')
+
+ -- CSV files in /tmp
+ localfiles(baseDir => '/tmp',
+ filter => '*.csv',
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(baseDir => '/tmp',
+ files => ARRAY['a.csv', 'b.csv'],
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(files => ARRAY['/tmp/a.csv', '/tmp/b.csv'],
+ format => 'csv')
+```
-#### CSV Format
+#### `S3` Function
-Use the `csv` format to read from CSV. This choice selects the Druid `CsvInputFormat` class.
+The `S3` table function represents the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) which reads
+files from an S3 bucket. The function accepts the following parameters to specify the
+objects to read:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+* `uris` (`ARRAY` of `VARCHAR`)
+* `prefix` (`VARCHAR`) - Corresponds to the JSON `prefixes` property, but allows a single
+ prefix.
+* `bucket` (`VARCHAR`) - Corresponds to the `bucket` field of the `objects` JSON field. SQL
+ does not have syntax for an array of objects. Instead, this function taks a single bucket,
+ and one or more objects within that bucket.
+* `paths` (`ARRAY` of `VARCHAR`) - Corresponds to the `path` fields of the `object` JSON field.
+ All paths are within the single `bucket` parameter.
-MSQ does not have the ability to infer schema from a CSV, file, so the `findColumnsFromHeader` property
-is unavailable. Instead, Columns are given using the `EXTEND` syntax described above.
+The S3 input source accepts one of the following patterns:
-#### Delimited Text Format
+* `uris` - A list of fully-qualified object URIs.
+* `prefixes` - A list of fully-qualified "folder" prefixes.
+* `bucket` and `paths` - A list of objects relative to the given bucket path.
+
+The `S3` function also accepts the following security properties:
-Use the `tsv` format to read from an arbitrary delimited (CSV-like) file such as tab-delimited,
-pipe-delimited, etc. This choice selects the Druid `DelimitedInputFormat` class.
+* `accessKeyId` (`VARCHAR`)
+* `secretAccessKey` (`VARCHAR`)
+* `assumeRoleArn` (`VARCHAR`)
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `delimiter` | The delimiter which separates fields. | `delimiter` | Yes |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+The `S3` table function does not support either the `clientConfig` or `proxyConfig`
+JSON properties.
-As noted above, MSQ cannot infer schema using headers. Use `EXTEND` instead.
+If you need the full power of the S3 input source, then consider the use of the `extern`
+function, which accepts the full S3 input source serializd as JSON. Alternatively,
+create a catalog external table that has the full set of properties, leaving just the
+`uris` or `paths` to be provided at query time.
+
+Examples, each of which correspond to an example on the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) page.
+The examples omit the format and schema; however you must remember to provide those
+in an actual query.
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json']))
+```
+
+```sql
+ TABLE(S3(prefixes => ARRAY['s3://foo/bar/', 's3://bar/foo/']))
+```
+
+```sql
+ -- Not an exact match for the JSON example: the S3 function allows
+ -- only one bucket.
+ TABLE(S3(bucket => 's3://foo`,
+ paths => ARRAY['bar/file1.json', 'foo/file2.json']))
+```
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json'],
+ accessKeyId => 'KLJ78979SDFdS2',
+ secretAccessKey => 'KLS89s98sKJHKJKJH8721lljkd'))
+```
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json'],
+ accessKeyId => 'KLJ78979SDFdS2',
+ secretAccessKey => 'KLS89s98sKJHKJKJH8721lljkd',
+ assumeRoleArn => 'arn:aws:iam::2981002874992:role/role-s3'))
+```
+
+#### Input Format
+
+Each of the table functions above requires that you specify a format using the `format`
+parameter which accepts a value the same as the format names used for `EXTERN` and described
+for [each input source](../ingestion/native-batch-input-sources.md).
+
+#### CSV Format
+
+The `csv` format selects the [CSV input format](../ingestion/data-formats.md#csv).
+Parameters:
+
+* `listDelimiter` (`VARCHAR`)
Review Comment:
Might be good to provide some examples for these different input format types, especially the ones where the user can provide parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066139789
##########
docs/multi-stage-query/reference.md:
##########
@@ -88,93 +118,260 @@ FROM TABLE(
http(
userName => 'bob',
password => 'secret',
- uris => 'http:foo.com/bar.csv',
+ uris => ARRAY['http:example.com/foo.csv', 'http:example.com/bar.csv'],
format => 'csv'
)
) (x VARCHAR, y VARCHAR, z BIGINT)
```
+#### Function Arguments
-The set of table functions and formats is preliminary in this release.
+These table functions are intended for use with the SQL by-name argument syntax
+as shown above. Because the functions include all parameters for all formats,
+using positional calls is both cumbersome and error-prone.
+
+Function argument names are generally the same as the JSON field names, except
+as noted below. Each argument has a SQL type which matches the JSON type. For
+arguments that take a string list in JSON, use the SQL `ARRAY[...]` syntax in
+SQL as shown in the above example.
+
+Array parameters are good candidates for use in parameterized queries. That is:
+
+```sql
+SELECT
+ <column>
+FROM TABLE(
+ http(
+ userName => 'bob',
+ password => 'secret',
+ uris => ?,
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR, z BIGINT)
+```
-#### `HTTP`
+Provide the list of URIs (in this case) as a query parameter in each ingest. Doing
+so is simpler than writing a script to insert the array into the SQL text.
-The `HTTP` table function represents the `HttpInputSource` class in Druid which allows you to
-read from an HTTP server. The function accepts the following arguments:
+#### `HTTP` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `userName` | Basic authentication user name | `httpAuthenticationUsername` | No |
-| `password` | Basic authentication password | `httpAuthenticationPassword` | No |
-| `passwordEnvVar` | Environment variable that contains the basic authentication password| `httpAuthenticationPassword` | No |
-| `uris` | Comma-separated list of URIs to read. | `uris` | Yes |
+The `HTTP` table function represents the
+[HTTP input source](../ingestion/native-batch-input-sources.md#http-input-source)
+to read from an HTTP server. The function accepts the following arguments:
-#### `INLINE`
+* `userName` (`VARCHAR`) - Same as JSON `httpAuthenticationUsername`.
+* `password` (`VARCHAR`) - Same as`httpAuthenticationPassword` when used with the default option.
+* `passwordEnvVar` (`VARCHAR`) - Same as the HTTP `httpAuthenticationPassword` when used with
+ the `"type": "environment"` option.
+* `uris` (`ARRAY` of `VARCHAR`)
-The `INLINE` table function represents the `InlineInputSource` class in Druid which provides
-data directly in the table function. The function accepts the following arguments:
+#### `INLINE` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `data` | Text lines of inline data. Separate lines with a newline. | `data` | Yes |
+The `INLINE` table function represents the
+[Inline input source](../ingestion/native-batch-input-sources.md#inline-input-source)
+which provides data directly in the table function. Parameter:
-#### `LOCALFILES`
+* `data` (`ARRAY` of `VARCHAR`) - Data lines, without a trailing newline, as an array.
-The `LOCALFILES` table function represents the `LocalInputSource` class in Druid which reads
+Example:
+
+```sql
+FROM TABLE(
+ inline(
+ data => ARRAY[
+ "a,b",
+ "c,d"],
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR)
+```
+
+
+#### `LOCALFILES` Function
+
+The `LOCALFILES` table function represents the
+[Local input source](../ingestion/native-batch-input-sources.md#local-input-source) which reads
files from the file system of the node running Druid. This is most useful for single-node
-installations. The function accepts the following arguments:
+installations. The function accepts the following parameters:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `baseDir` | Directory to read from. | `baseDir` | No |
-| `filter` | Filter pattern to read. Example: `*.csv`. | `filter` | No |
-| `files` | Comma-separated list of files to read. | `files` | No |
+* `baseDir`
+* `filter`
+* `files`
-You must either provide the `baseDir` or the list of `files`. You can provide both, in which case
-the files are assumed relative to the `baseDir`. If you provide a `filter`, you must provide the
-`baseDir`.
+When the local files input source is used directly in an `extern` function, or ingestion spec, you
+can provide either `baseDir` and `filter` or `files` but not both. This function, however, allows
+you to provide any of the following combinations:
-#### Table Function Format
+* `baseDir` - Matches all files in the given directory. (Assumes the filter is `*`.)
+* `baseDir` and `filter` - Match files in the given directory using the filter.
+* `baseDir` and `files` - A set of files relative to `baseDir`.
+* `files` - The files should be absolute paths, else they will be computed relative to Druid's
+ working directory (usually the Druid install directory.)
-Each of the table functions above requires that you specify a format.
+Examples:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `format` | The input format, using the same names as for `EXTERN`. | `inputFormat.type` | Yes |
+```sql
+ -- All files in /tmp, which must be CSV files
+ localfiles(baseDir => '/tmp',
+ format => 'csv')
+
+ -- CSV files in /tmp
+ localfiles(baseDir => '/tmp',
+ filter => '*.csv',
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(baseDir => '/tmp',
+ files => ARRAY['a.csv', 'b.csv'],
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(files => ARRAY['/tmp/a.csv', '/tmp/b.csv'],
+ format => 'csv')
+```
-#### CSV Format
+#### `S3` Function
-Use the `csv` format to read from CSV. This choice selects the Druid `CsvInputFormat` class.
+The `S3` table function represents the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) which reads
+files from an S3 bucket. The function accepts the following parameters to specify the
+objects to read:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+* `uris` (`ARRAY` of `VARCHAR`)
+* `prefix` (`VARCHAR`) - Corresponds to the JSON `prefixes` property, but allows a single
+ prefix.
+* `bucket` (`VARCHAR`) - Corresponds to the `bucket` field of the `objects` JSON field. SQL
+ does not have syntax for an array of objects. Instead, this function taks a single bucket,
+ and one or more objects within that bucket.
+* `paths` (`ARRAY` of `VARCHAR`) - Corresponds to the `path` fields of the `object` JSON field.
Review Comment:
Should we add the `accessKeyId`, `secretAccessKey`, and `assumeRoleArn` here as some of the examples below use them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066287847
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
Review Comment:
nit: why it this a property while the others are fields?
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECTS_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
+ private static final List<ParameterDefn> SECURITY_PARAMS = Arrays.asList(
+ new Parameter(ACCESS_KEY_ID_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(SECRET_ACCESS_KEY_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(ASSUME_ROLE_ARN_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the S3InputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PREFIXES_FIELD = "prefixes";
+ private static final String OBJECTS_FIELD = "objects";
+ private static final String OBJECT_GLOB_FIELD = "objectGlob";
+ private static final String PROPERTIES_FIELD = "properties";
+ private static final String ACCESS_KEY_ID_FIELD = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY_FIELD = "secretAccessKey";
+ private static final String ASSUME_ROLE_ARN_FIELD = "assumeRoleArn";
+
+ @Override
+ public String typeValue()
+ {
+ return S3StorageDruidModule.SCHEME;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return S3InputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (hasFormat && !hasColumns) {
+ throw new IAE(
+ "An external S3 table with a format must also provide the corresponding columns"
+ );
+ }
+
+ // The user can either provide a bucket, or can provide one of the valid items.
+ final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
+ final boolean hasBucket = bucket != null;
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
+ final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
+ final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
+ final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
+ if (hasBucket) {
+ if (hasUris || hasPrefix || hasObjects) {
+ throw new IAE(
+ "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
+ BUCKET_PROPERTY,
+ URIS_FIELD,
+ PREFIXES_FIELD,
+ OBJECTS_FIELD
+ );
+ }
+ if (hasGlob) {
+ throw new IAE(
+ "The %s property cannot be provided when the the %s property is set",
+ OBJECT_GLOB_FIELD,
+ BUCKET_PROPERTY
+ );
+ }
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ sourceMap.put(URIS_FIELD, Collections.singletonList(bucket));
Review Comment:
Where does validation fail if we dont patch in dummy URI here?
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECTS_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
Review Comment:
should this be `OBJECT_GLOB_PARAM_DEFN`, since its singular?
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import org.apache.druid.catalog.model.CatalogUtils;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.BaseTableFunction.Parameter;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterType;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.s3.S3InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.storage.s3.S3StorageDruidModule;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Catalog definition for the S3 input source.
+ * <p>
+ * The catalog entry contains a serialized S3 input source, with some simplifying variations.
+ * The catalog entry can define the {@code bucket} table property which is the (single) bucket
+ * value to use when creating the list of objects: the catalog provides the bucket, the table
+ * function provides the list of objects. (Done this way since defining two correlated lists
+ * in SQL is awkward, and credentials make the most sense when working with a single bucket.)
+ * <p>
+ * The ad-hoc function allows the various ways to specify the objects, but not the configuration
+ * parameters. If the user wishes to use such parameters, they should be defined in a catalog
+ * entry, since providing maps in SQL is awkward.
+ * <p>
+ * The partial table function can be of various forms:
+ * <ul>
+ * <li>Fully define the table, which means providing the full set of S3 properties and not
+ * providing the table-level {@code bucket} property. This form is complete and does't need
+ * a table function. If used with a table function, the function provides the {@code glob}
+ * parameter (if not already provided in the table spec.)</li>
+ * <li>Partially define the table: using URIs with the {@code glob} to be provided later, or
+ * by using the {@code bucket} table property. The table function provides the {@code objects}
+ * parameter to specify the specific objects. This form provides both the format and the list
+ * of columns.</li>
+ * <li>Partially define the table as a connection: provide only the {@code bucket} property,
+ * and omit both the format and the columns. The table function requests the {@code objects}
+ * and the {@code format}. The user must provide the list of columns.</li>
+ * </ul>
+ *
+ * @see {@link S3InputSource} for details on the meaning of the various properties, and the rules
+ * about valid combinations
+ */
+public class S3InputSourceDefn extends FormattedInputSourceDefn
+{
+ public static final String URIS_PARAMETER = "uris";
+ public static final String PREFIXES_PARAMETER = "prefixes";
+ public static final String BUCKET_PARAMETER = "bucket";
+ public static final String PATHS_PARAMETER = "paths";
+ public static final String ACCESS_KEY_ID_PARAMETER = "accessKeyId";
+ public static final String SECRET_ACCESS_KEY_PARAMETER = "secretAccessKey";
+ public static final String ASSUME_ROLE_ARN_PARAMETER = "assumeRoleArn";
+
+ /**
+ * The {@code objectGlob} property exists in S3, but is not documented. The corresponding
+ * function parameter also exists, but is not documented.
+ */
+ public static final String OBJECT_GLOB_PARAMETER = "objectGlob";
+
+ public static final String BUCKET_PROPERTY = "bucket";
+
+ private static final ParameterDefn URI_PARAM_DEFN = new Parameter(URIS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn PREFIXES_PARAM_DEFN = new Parameter(PREFIXES_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn BUCKET_PARAM_DEFN = new Parameter(BUCKET_PARAMETER, ParameterType.VARCHAR, true);
+ private static final ParameterDefn PATHS_PARAM_DEFN = new Parameter(PATHS_PARAMETER, ParameterType.VARCHAR_ARRAY, true);
+ private static final ParameterDefn OBJECTS_GLOB_PARAM_DEFN = new Parameter(OBJECT_GLOB_PARAMETER, ParameterType.VARCHAR, true);
+ private static final List<ParameterDefn> SECURITY_PARAMS = Arrays.asList(
+ new Parameter(ACCESS_KEY_ID_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(SECRET_ACCESS_KEY_PARAMETER, ParameterType.VARCHAR, true),
+ new Parameter(ASSUME_ROLE_ARN_PARAMETER, ParameterType.VARCHAR, true)
+ );
+
+ // Field names in the S3InputSource
+ private static final String URIS_FIELD = "uris";
+ private static final String PREFIXES_FIELD = "prefixes";
+ private static final String OBJECTS_FIELD = "objects";
+ private static final String OBJECT_GLOB_FIELD = "objectGlob";
+ private static final String PROPERTIES_FIELD = "properties";
+ private static final String ACCESS_KEY_ID_FIELD = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY_FIELD = "secretAccessKey";
+ private static final String ASSUME_ROLE_ARN_FIELD = "assumeRoleArn";
+
+ @Override
+ public String typeValue()
+ {
+ return S3StorageDruidModule.SCHEME;
+ }
+
+ @Override
+ protected Class<? extends InputSource> inputSourceClass()
+ {
+ return S3InputSource.class;
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ final boolean hasFormat = table.inputFormatMap != null;
+ final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());
+
+ if (hasFormat && !hasColumns) {
+ throw new IAE(
+ "An external S3 table with a format must also provide the corresponding columns"
+ );
+ }
+
+ // The user can either provide a bucket, or can provide one of the valid items.
+ final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
+ final boolean hasBucket = bucket != null;
+ final Map<String, Object> sourceMap = table.inputSourceMap;
+ final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
+ final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
+ final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
+ final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
+ if (hasBucket) {
+ if (hasUris || hasPrefix || hasObjects) {
+ throw new IAE(
+ "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
+ BUCKET_PROPERTY,
+ URIS_FIELD,
+ PREFIXES_FIELD,
+ OBJECTS_FIELD
+ );
+ }
+ if (hasGlob) {
+ throw new IAE(
+ "The %s property cannot be provided when the the %s property is set",
+ OBJECT_GLOB_FIELD,
+ BUCKET_PROPERTY
+ );
+ }
+
+ // Patch in a dummy URI so that validation of the rest of the fields
+ // will pass.
+ sourceMap.put(URIS_FIELD, Collections.singletonList(bucket));
+ }
+ super.validate(table);
+ }
+
+ @Override
+ protected List<ParameterDefn> adHocTableFnParameters()
+ {
+ return CatalogUtils.concatLists(
+ Arrays.asList(
+ URI_PARAM_DEFN,
+ PREFIXES_PARAM_DEFN,
+ BUCKET_PARAM_DEFN,
+ PATHS_PARAM_DEFN,
+ OBJECTS_GLOB_PARAM_DEFN
+ ),
+ SECURITY_PARAMS
+ );
+ }
+
+ @Override
+ protected void convertArgsToSourceMap(Map<String, Object> jsonMap, Map<String, Object> args)
+ {
+ jsonMap.put(InputSource.TYPE_PROPERTY, S3StorageDruidModule.SCHEME);
+ final List<String> uris = CatalogUtils.getStringArray(args, URIS_PARAMETER);
+ final List<String> prefixes = CatalogUtils.getStringArray(args, PREFIXES_PARAMETER);
+ final String bucket = CatalogUtils.getNonBlankString(args, BUCKET_PARAMETER);
+ final List<String> paths = CatalogUtils.getStringArray(args, PATHS_PARAMETER);
+ final String objectGlob = CatalogUtils.getNonBlankString(args, OBJECT_GLOB_PARAMETER);
+ final boolean hasUris = uris != null;
+ final boolean hasPrefixes = prefixes != null;
+ final boolean hasBucket = bucket != null;
+ final boolean hasPaths = !CollectionUtils.isNullOrEmpty(paths);
+ if (hasPaths && !hasBucket) {
+ throw new IAE(
+ "S3 requires the %s parameter if %s is set",
+ BUCKET_PARAMETER,
+ PATHS_PARAMETER
+ );
+ }
+ if ((hasUris && (hasPrefixes || hasBucket)) || (hasPrefixes && hasBucket)) {
+ throw new IAE(
+ "S3 accepts only one of %s, %s or %s",
+ PATHS_PARAMETER,
+ BUCKET_PARAMETER,
+ PREFIXES_PARAMETER
+ );
+ }
+ if (!hasUris && !hasPrefixes && !hasBucket) {
+ throw new IAE(
+ "S3 requires one of %s, %s or %s",
+ PATHS_PARAMETER,
+ BUCKET_PARAMETER,
+ PREFIXES_PARAMETER
+ );
+ }
+ if (hasUris) {
+ jsonMap.put(URIS_FIELD, CatalogUtils.stringListToUriList(uris));
+ }
+ if (hasPrefixes) {
+ jsonMap.put(PREFIXES_FIELD, prefixes);
+ }
+ if (hasBucket) {
+ if (!hasPaths) {
+ throw new IAE("When using the %s parameter, %s must also be provided", BUCKET_PARAMETER, PATHS_PARAMETER);
+ }
+ List<CloudObjectLocation> objects = new ArrayList<>();
+ for (String obj : paths) {
+ objects.add(new CloudObjectLocation(bucket, obj));
+ }
+ jsonMap.put(OBJECTS_FIELD, objects);
+ }
+ if (objectGlob != null) {
+ jsonMap.put(OBJECT_GLOB_FIELD, objectGlob);
Review Comment:
can you have both objects and object glob?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066584622
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
Review Comment:
This this be moved into the `AdHocTableFunction` class, since it seems it should only be used in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066343900
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -93,15 +100,54 @@ public static <T> T safeCast(Object value, Class<T> type, String propertyName)
catch (ClassCastException e) {
throw new IAE("Value [%s] is not valid for property %s, expected type %s",
value,
- propertyName,
+ key,
type.getSimpleName()
);
}
}
- public static <T> T safeGet(Map<String, Object> map, String propertyName, Class<T> type)
+ public static <T> T safeGet(Map<String, Object> map, String key, Class<T> type)
{
- return safeCast(map.get(propertyName), type, propertyName);
+ return safeCast(map.get(key), type, key);
+ }
+
+ public static long getLong(Map<String, Object> map, String key)
Review Comment:
Should we catch any `ClassCastException` here as is being done in `safeGet` and return more friendly error to the user in the case of error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1068789221
##########
server/src/main/java/org/apache/druid/catalog/model/table/ExternalTableSpec.java:
##########
@@ -28,13 +28,13 @@
/**
* Catalog form of an external table specification used to pass along the three
* components needed for an external table in MSQ ingest. Just like
- * {@code ExternalTableSource}, except that the parameters are not required
- * to be non-null.
+ * {@code ExternalTableSource}, except that the class is visible in this
+ * module.
*/
public class ExternalTableSpec
{
- @Nullable public final InputSource inputSource;
- @Nullable public final InputFormat inputFormat;
+ public final InputSource inputSource;
Review Comment:
Yes. That's the implication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on PR #13627:
URL: https://github.com/apache/druid/pull/13627#issuecomment-1371761837
To anyone who wants to review, here is a summary of an overview of this PR provided via a private channel.
Let's assume you are already familiar with the model layer from last time. The key changes here are to make table functions more generic. First, there is a table function abstraction: something that has a set of parameter definitions, accepts a set of arguments, and returns an external table definition. The table function replaces the property annotations from the prior PR, so the annotation stuff was removed. Before, there was a definition for each kind of external table. This was too restrictive. So, now there is a generic external table definition that has two "parts". One is a definition of the input source, the other is a list of possible input formats.
Each input source has its own metadata class which says which properties are available (as before), but also says what table functions are available. There three kind of functions. 1) An "ad-hoc" (from scratch) function that is a fancy form of the existing extern function. 2) a "partial" function where some of the data resides in the catalog, the rest is given by the query using the table name as a function name. 3) a "complete" function where the catalog table name can be used either as a table or as a zero-argument table function. The input source metadata specifies the various properties, how function arguments are mapped to properties, and how to merge catalog and argument values. Formats are similar, but are shared by all input sources.
Given this, the second part is the SQL layer table functions are redone to use the table function abstraction and the input source definition. Basically, the HTTP input source function holds onto the HTTP input source definition, and asks that definition for the table function definition, which then provide the list of parameters and process the actual call. Given that, the related code is kind of one big thing: it won't work to use, say, the old Calcite code with the new model code or visa-versa.
You can start here: `server/src/main/java/org/apache/druid/catalog/model/table`. That's the core. Then, spiral outwards. `server/src/main/java/org/apache/druid/catalog/model` has a bunch of supporting changes. `sql/src/main/java/org/apache/druid/sql/calcite/external` is the Calcite integration for the table functions. Most of this stuff is either a complete mystery (because you don't know the bizarre ways that Calcite handles table macros) or trivial (because it builds on the code mentioned above.) Another big wad of files are the corresponding tests.
Oh, and if you want to know what all this code actually does, look at the `reference.md` documentation file.
A follow-on PR will add catalog integration to the Calcite planner so that we incorporate catalog information when planning an MSQ query. There are some bits toward that goal in this PR (where they overlap with external tables). However, to keep this PR from growing bigger; the planner work, which uses table specs from the catalog, comes later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066355611
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
Should this return the enum name of the Period string? If the user provides the period name on the next go, that seems that it will fail the validation here, as it wont be a valid Period?
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
Should this return the enum name as it is or the Period string? If the user provides the period name on the next go, that seems that it will fail the validation here, as it wont be a valid Period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066355611
##########
server/src/main/java/org/apache/druid/catalog/model/CatalogUtils.java:
##########
@@ -144,4 +188,125 @@ public static <T> List<T> concatLists(
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
+
+ /**
+ * Get a string parameter that can either be null or non-blank.
+ */
+ public static String getNonBlankString(Map<String, Object> args, String parameter)
+ {
+ String value = CatalogUtils.getString(args, parameter);
+ if (value != null) {
+ value = value.trim();
+ if (value.isEmpty()) {
+ throw new IAE("%s parameter cannot be a blank string", parameter);
+ }
+ }
+ return value;
+ }
+
+ public static List<String> getUriListArg(Map<String, Object> args, String parameter)
+ {
+ String urisString = CatalogUtils.getString(args, parameter);
+ if (Strings.isNullOrEmpty(urisString)) {
+ throw new IAE("One or more values are required for parameter %s", parameter);
+ }
+ return stringToList(urisString);
+ }
+
+ public static List<URI> stringToUriList(String uris)
+ {
+ return stringListToUriList(stringToList(uris));
+ }
+
+ /**
+ * Convert a list of strings to a list of {@link URI} objects.
+ */
+ public static List<URI> stringListToUriList(List<String> list)
+ {
+ if (list == null) {
+ return null;
+ }
+ List<URI> uris = new ArrayList<>();
+ for (String strValue : list) {
+ try {
+ uris.add(new URI(strValue));
+ }
+ catch (URISyntaxException e) {
+ throw new IAE(StringUtils.format("Argument [%s] is not a valid URI", strValue));
+ }
+ }
+ return uris;
+ }
+
+ /**
+ * Merge the properties for an object using a set of updates in a map. If the
+ * update value is {@code null}, then remove the property in the revised set. If the
+ * property is known, use the column definition to merge the values. Else, the
+ * update replaces any existing value.
+ * <p>
+ * This method does not validate the properties, except as needed to do a
+ * merge. A separate validation step is done on the final, merged object.
+ */
+ public static Map<String, Object> mergeProperties(
+ final Map<String, PropertyDefn<?>> properties,
+ final Map<String, Object> source,
+ final Map<String, Object> update
+ )
+ {
+ if (update == null) {
+ return source;
+ }
+ if (source == null) {
+ return update;
+ }
+ final Map<String, Object> merged = new HashMap<>(source);
+ for (Map.Entry<String, Object> entry : update.entrySet()) {
+ if (entry.getValue() == null) {
+ merged.remove(entry.getKey());
+ } else {
+ Object value = entry.getValue();
+ final PropertyDefn<?> propDefn = properties.get(entry.getKey());
+ if (propDefn != null) {
+ value = propDefn.merge(merged.get(entry.getKey()), entry.getValue());
+ }
+ merged.put(entry.getKey(), value);
+ }
+ }
+ return merged;
+ }
+
+ public static void validateGranularity(String value)
+ {
+ if (value == null) {
+ return;
+ }
+ Granularity granularity;
+ try {
+ granularity = new PeriodGranularity(new Period(value), null, null);
+ }
+ catch (IllegalArgumentException e) {
+ throw new IAE(StringUtils.format("[%s] is an invalid granularity string", value));
+ }
+ if (!GranularityType.isStandard(granularity)) {
+ throw new IAE(
+ "Unsupported segment graularity. "
+ + "Please use an equivalent of these granularities: %s.",
+ Arrays.stream(GranularityType.values())
+ .filter(granularityType -> !granularityType.equals(GranularityType.NONE))
+ .map(Enum::name)
Review Comment:
Should this return the enum name as it is or the Period string (ex. this would return `MINUTE`, or `PT1M`)? If the user provides the period name on the next go, that seems that it will fail the validation here, as it wont be a valid Period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066564475
##########
server/src/main/java/org/apache/druid/catalog/model/TableDefn.java:
##########
@@ -99,26 +93,13 @@ public void validateColumns(List<ColumnSpec> columns, ObjectMapper jsonMapper)
if (!names.add(colSpec.name())) {
throw new IAE("Duplicate column name: " + colSpec.name());
}
- ColumnDefn.ResolvedColumn resolvedCol = resolveColumn(colSpec);
- resolvedCol.validate(jsonMapper);
+ colSpec.validate();
+ validateColumn(colSpec);
}
}
- /**
- * Resolve the column type to produce a composite object that holds
- * both the definition and the column spec.
- */
- public ColumnDefn.ResolvedColumn resolveColumn(ColumnSpec spec)
+ protected void validateColumn(ColumnSpec colSpec)
Review Comment:
I assume this will be defined at some later time? what will this function do, javadoc would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066584727
##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.Columns;
+import org.apache.druid.catalog.model.TableDefnRegistry;
+import org.apache.druid.data.input.InputFormat;
+import org.apache.druid.data.input.InputSource;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for input source definitions.
+ *
+ * @see {@link FormattedInputSourceDefn} for the base class for (most) input formats
+ * which take an input format.
+ */
+public abstract class BaseInputSourceDefn implements InputSourceDefn
+{
+ /**
+ * The "from-scratch" table function for this input source. The parameters
+ * are those defined by the subclass, and the apply simply turns around and
+ * asks the input source definition to do the conversion.
+ */
+ public class AdHocTableFunction extends BaseTableFunction
+ {
+ public AdHocTableFunction(List<ParameterDefn> parameters)
+ {
+ super(parameters);
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ requireSchema(fnName, columns);
+ return convertArgsToTable(args, columns, jsonMapper);
+ }
+ }
+
+ /**
+ * The "partial" table function that starts with a catalog external table spec, then
+ * uses SQL function arguments to "complete" (i.e. fill in) the missing properties to
+ * produce a complete table which is then converted to an external table which Calcite
+ * can use.
+ * <p>
+ * The set of parameters depends on the input source and on whether or not the catalog
+ * spec provides a format.
+ */
+ public class PartialTableFunction extends BaseTableFunction
+ {
+ private final ResolvedExternalTable table;
+
+ public PartialTableFunction(final ResolvedExternalTable table, List<ParameterDefn> params)
+ {
+ super(params);
+ this.table = table;
+ }
+
+ @Override
+ public ExternalTableSpec apply(
+ final String fnName,
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ if (CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns())) {
+ requireSchema(fnName, columns);
+ }
+ return convertCompletedTable(table, args, columns);
+ }
+ }
+
+ /**
+ * The one and only from-scratch table function for this input source. The
+ * function is defined a bind time, not construction time, since it typically
+ * needs visibility to the set of available input formats.
+ */
+ private AdHocTableFunction adHocTableFn;
+
+ /**
+ * Overridden by each subclass to return the input source class to be
+ * used for JSON conversions.
+ */
+ protected abstract Class<? extends InputSource> inputSourceClass();
+
+ @Override
+ public void bind(TableDefnRegistry registry)
+ {
+ this.adHocTableFn = defineAdHocTableFunction();
+ }
+
+ @Override
+ public void validate(ResolvedExternalTable table)
+ {
+ convertTableToSource(table);
+ }
+
+ /**
+ * Overridden by each subclass to define the parameters needed by each
+ * input source.
+ */
+ protected abstract AdHocTableFunction defineAdHocTableFunction();
+
+ @Override
+ public TableFunction adHocTableFn()
+ {
+ return adHocTableFn;
+ }
+
+ /**
+ * Define a table "from scratch" using SQL function arguments.
+ */
+ protected ExternalTableSpec convertArgsToTable(
+ final Map<String, Object> args,
+ final List<ColumnSpec> columns,
+ final ObjectMapper jsonMapper
+ )
+ {
+ return new ExternalTableSpec(
+ convertArgsToSource(args, jsonMapper),
+ convertArgsToFormat(args, columns, jsonMapper),
+ Columns.convertSignature(columns)
+ );
+ }
+
+ /**
+ * Convert the input source using arguments to a "from scratch" table function.
+ */
+ protected InputSource convertArgsToSource(Map<String, Object> args, ObjectMapper jsonMapper)
Review Comment:
This this be moved into the `AdHocTableFunction` class, since it seems it should only be used in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1066140953
##########
docs/multi-stage-query/reference.md:
##########
@@ -88,93 +118,260 @@ FROM TABLE(
http(
userName => 'bob',
password => 'secret',
- uris => 'http:foo.com/bar.csv',
+ uris => ARRAY['http:example.com/foo.csv', 'http:example.com/bar.csv'],
format => 'csv'
)
) (x VARCHAR, y VARCHAR, z BIGINT)
```
+#### Function Arguments
-The set of table functions and formats is preliminary in this release.
+These table functions are intended for use with the SQL by-name argument syntax
+as shown above. Because the functions include all parameters for all formats,
+using positional calls is both cumbersome and error-prone.
+
+Function argument names are generally the same as the JSON field names, except
+as noted below. Each argument has a SQL type which matches the JSON type. For
+arguments that take a string list in JSON, use the SQL `ARRAY[...]` syntax in
+SQL as shown in the above example.
+
+Array parameters are good candidates for use in parameterized queries. That is:
+
+```sql
+SELECT
+ <column>
+FROM TABLE(
+ http(
+ userName => 'bob',
+ password => 'secret',
+ uris => ?,
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR, z BIGINT)
+```
-#### `HTTP`
+Provide the list of URIs (in this case) as a query parameter in each ingest. Doing
+so is simpler than writing a script to insert the array into the SQL text.
-The `HTTP` table function represents the `HttpInputSource` class in Druid which allows you to
-read from an HTTP server. The function accepts the following arguments:
+#### `HTTP` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `userName` | Basic authentication user name | `httpAuthenticationUsername` | No |
-| `password` | Basic authentication password | `httpAuthenticationPassword` | No |
-| `passwordEnvVar` | Environment variable that contains the basic authentication password| `httpAuthenticationPassword` | No |
-| `uris` | Comma-separated list of URIs to read. | `uris` | Yes |
+The `HTTP` table function represents the
+[HTTP input source](../ingestion/native-batch-input-sources.md#http-input-source)
+to read from an HTTP server. The function accepts the following arguments:
-#### `INLINE`
+* `userName` (`VARCHAR`) - Same as JSON `httpAuthenticationUsername`.
+* `password` (`VARCHAR`) - Same as`httpAuthenticationPassword` when used with the default option.
+* `passwordEnvVar` (`VARCHAR`) - Same as the HTTP `httpAuthenticationPassword` when used with
+ the `"type": "environment"` option.
+* `uris` (`ARRAY` of `VARCHAR`)
-The `INLINE` table function represents the `InlineInputSource` class in Druid which provides
-data directly in the table function. The function accepts the following arguments:
+#### `INLINE` Function
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `data` | Text lines of inline data. Separate lines with a newline. | `data` | Yes |
+The `INLINE` table function represents the
+[Inline input source](../ingestion/native-batch-input-sources.md#inline-input-source)
+which provides data directly in the table function. Parameter:
-#### `LOCALFILES`
+* `data` (`ARRAY` of `VARCHAR`) - Data lines, without a trailing newline, as an array.
-The `LOCALFILES` table function represents the `LocalInputSource` class in Druid which reads
+Example:
+
+```sql
+FROM TABLE(
+ inline(
+ data => ARRAY[
+ "a,b",
+ "c,d"],
+ format => 'csv'
+ )
+ ) (x VARCHAR, y VARCHAR)
+```
+
+
+#### `LOCALFILES` Function
+
+The `LOCALFILES` table function represents the
+[Local input source](../ingestion/native-batch-input-sources.md#local-input-source) which reads
files from the file system of the node running Druid. This is most useful for single-node
-installations. The function accepts the following arguments:
+installations. The function accepts the following parameters:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `baseDir` | Directory to read from. | `baseDir` | No |
-| `filter` | Filter pattern to read. Example: `*.csv`. | `filter` | No |
-| `files` | Comma-separated list of files to read. | `files` | No |
+* `baseDir`
+* `filter`
+* `files`
-You must either provide the `baseDir` or the list of `files`. You can provide both, in which case
-the files are assumed relative to the `baseDir`. If you provide a `filter`, you must provide the
-`baseDir`.
+When the local files input source is used directly in an `extern` function, or ingestion spec, you
+can provide either `baseDir` and `filter` or `files` but not both. This function, however, allows
+you to provide any of the following combinations:
-#### Table Function Format
+* `baseDir` - Matches all files in the given directory. (Assumes the filter is `*`.)
+* `baseDir` and `filter` - Match files in the given directory using the filter.
+* `baseDir` and `files` - A set of files relative to `baseDir`.
+* `files` - The files should be absolute paths, else they will be computed relative to Druid's
+ working directory (usually the Druid install directory.)
-Each of the table functions above requires that you specify a format.
+Examples:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `format` | The input format, using the same names as for `EXTERN`. | `inputFormat.type` | Yes |
+```sql
+ -- All files in /tmp, which must be CSV files
+ localfiles(baseDir => '/tmp',
+ format => 'csv')
+
+ -- CSV files in /tmp
+ localfiles(baseDir => '/tmp',
+ filter => '*.csv',
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(baseDir => '/tmp',
+ files => ARRAY['a.csv', 'b.csv'],
+ format => 'csv')
+
+ -- /tmp/a.csv and /tmp/b.csv
+ localfiles(files => ARRAY['/tmp/a.csv', '/tmp/b.csv'],
+ format => 'csv')
+```
-#### CSV Format
+#### `S3` Function
-Use the `csv` format to read from CSV. This choice selects the Druid `CsvInputFormat` class.
+The `S3` table function represents the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) which reads
+files from an S3 bucket. The function accepts the following parameters to specify the
+objects to read:
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+* `uris` (`ARRAY` of `VARCHAR`)
+* `prefix` (`VARCHAR`) - Corresponds to the JSON `prefixes` property, but allows a single
+ prefix.
+* `bucket` (`VARCHAR`) - Corresponds to the `bucket` field of the `objects` JSON field. SQL
+ does not have syntax for an array of objects. Instead, this function taks a single bucket,
+ and one or more objects within that bucket.
+* `paths` (`ARRAY` of `VARCHAR`) - Corresponds to the `path` fields of the `object` JSON field.
+ All paths are within the single `bucket` parameter.
-MSQ does not have the ability to infer schema from a CSV, file, so the `findColumnsFromHeader` property
-is unavailable. Instead, Columns are given using the `EXTEND` syntax described above.
+The S3 input source accepts one of the following patterns:
-#### Delimited Text Format
+* `uris` - A list of fully-qualified object URIs.
+* `prefixes` - A list of fully-qualified "folder" prefixes.
+* `bucket` and `paths` - A list of objects relative to the given bucket path.
+
+The `S3` function also accepts the following security properties:
-Use the `tsv` format to read from an arbitrary delimited (CSV-like) file such as tab-delimited,
-pipe-delimited, etc. This choice selects the Druid `DelimitedInputFormat` class.
+* `accessKeyId` (`VARCHAR`)
+* `secretAccessKey` (`VARCHAR`)
+* `assumeRoleArn` (`VARCHAR`)
-| Name | Description | JSON equivalent | Required |
-| ---- | ----------- | --------------- | -------- |
-| `delimiter` | The delimiter which separates fields. | `delimiter` | Yes |
-| `listDelimiter` | The delimiter to use for fields that represent a list of strings. | `listDelimiter` | No |
-| `skipRows` | The number of rows to skip at the start of the file. Default is 0. | `skipHeaderRows` | No |
+The `S3` table function does not support either the `clientConfig` or `proxyConfig`
+JSON properties.
-As noted above, MSQ cannot infer schema using headers. Use `EXTEND` instead.
+If you need the full power of the S3 input source, then consider the use of the `extern`
+function, which accepts the full S3 input source serializd as JSON. Alternatively,
+create a catalog external table that has the full set of properties, leaving just the
+`uris` or `paths` to be provided at query time.
+
+Examples, each of which correspond to an example on the
+[S3 input source](../ingestion/native-batch-input-sources.md#s3-input-source) page.
+The examples omit the format and schema; however you must remember to provide those
+in an actual query.
+
+```sql
+ TABLE(S3(uris => ARRAY['s3://foo/bar/file.json', 's3://bar/foo/file2.json']))
Review Comment:
nit: to make consistent with the format of the localFiles function given above, should we drop the `TABLE(` prefix here, and specify the format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zachjsh commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
zachjsh commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067583226
##########
server/src/main/java/org/apache/druid/catalog/model/table/InputFormatDefn.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+import org.apache.druid.catalog.model.table.TableFunction.ParameterDefn;
+import org.apache.druid.data.input.InputFormat;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Metadata about a Druid {@link InputFormat}. Provides the logic to
+ * convert from a table spec or SQL function arguments to a specific
+ * form of {@code InputFormat}. There is one instance of this interface for each
+ * supported {@code InputFormat}.
+ */
+public interface InputFormatDefn
+{
+ /**
+ * @return the string used to identify the input format type in the serialized
+ * JSON for the input format. This is also the value used in the {@link format}
+ * property for SQL functions.
+ */
+ String typeValue();
+
+ /**
+ * Given a resolved table that has the serialized JSON converted to a Java map,
+ * validate the values of that map, typically by converting that map the target
+ * input format object (after adjustments and filling in dummy columns.) THe
+ * goal is to validate the information the user has provided in the table spec.
+ * The final format information is validated elsewhere.
+ */
+ void validate(ResolvedExternalTable table);
+
+ /**
+ * Obtain the parameters used to fully define an input format in a SQL function
+ * that defines an external table from scratch. Note that the final list of table
+ * function arguments combines parameters from all the various input sources. It
+ * is legal for multiple formats to define the same parameter, as long as both
+ * definitions are of the same type.
+ * @return
+ */
+ List<ParameterDefn> parameters();
+
+ /**
+ * Create an input format instance from the values provided as arguments that
+ * correspond to the defined parameters. The map provided is guaranteed to have
+ * a value for each parameter defined as non-optional, and those values will be
+ * of the type defined in the parameter. The map will also contain values for
+ * the input source: the format should values that don't correspond to parameters
Review Comment:
theres some type here, think trying to say that the format shouldn't have values that don't correspond to parameters that it defined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekrb19 commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
abhishekrb19 commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1067735870
##########
server/src/main/java/org/apache/druid/catalog/model/table/TableFunction.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.catalog.model.table;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.druid.catalog.model.ColumnSpec;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Representation of a SQL table function. The SQL module is not visible here.
+ * To avoid unnecessary dependencies, this class provides the information that
+ * Calcite needs, but without Calcite dependencies. Each function defines some
+ * number of arguments needed to create an external table. This same class
+ * can define a "from scratch" table, or to convert a partial table definition
+ * in the catalog into a full-defined external table.
+ * <p>
+ * The parameters tell Calcite the allowed parameters. Functions created from
+ * this class are most useful when used with named arguments:<code><pre>
+ * SELECT ... FROM TABLE(thisFn(foo -> "bar", answer -> 42))
+ * </pre</code>
+ * <p>
+ * Calcite provides actual arguments as an array, with null values for arguments
+ * which the user did not provided. That form is not helpful for this use case.
Review Comment:
typo: provide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] imply-cheddar commented on a diff in pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13627:
URL: https://github.com/apache/druid/pull/13627#discussion_r1068754168
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidCalciteSchemaModule.java:
##########
@@ -64,7 +64,7 @@ public void configure(Binder binder)
}
@Provides
- @Singleton
+ @LazySingleton
Review Comment:
`@Singleton` should NEVER be used. If it exists in any of our code we should remove it immediately. That's the reason. If anything it deserves a checkstyle validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers merged pull request #13627: Much improved table functions
Posted by GitBox <gi...@apache.org>.
paul-rogers merged PR #13627:
URL: https://github.com/apache/druid/pull/13627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org