You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "paul-rogers (via GitHub)" <gi...@apache.org> on 2023/02/09 05:44:34 UTC

[GitHub] [druid] paul-rogers opened a new pull request, #13780: Refinements to input-source specific table functions

paul-rogers opened a new pull request, #13780:
URL: https://github.com/apache/druid/pull/13780

   This is a follow-up to the earlier PR that introduced input-source specific table functions (e.g. `http`, `s3`, `localfiles`, etc.) This PR is mostly cleanup and bug fixes and is a spin-off of the effort to integrate the catalog with Calcite. These changes are done as a separate PR to keep each PR to a reasonable size.
   
   * Fixes various bugs
   * Improves the structure of the table function classes 
   * Adds unit and integration tests
   
   In particular, this PR fixes an issue with the feature that allows an SQL feature with the `extern` function.
   
   #### Release note
   
   No user-visible changes: just refinements to functionality already described in an earlier PR.
   
   <hr>
   
   This PR has:
   
   - [X] been self-reviewed.
   - [X] a release note entry in the PR description.
   - [X] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [X] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [X] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [X] added integration tests.
   - [X] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102132395


##########
server/src/main/java/org/apache/druid/catalog/model/table/FormattedInputSourceDefn.java:
##########
@@ -95,7 +95,10 @@ protected List<ParameterDefn> addFormatParameters(
   )
   {
     final List<ParameterDefn> toAdd = new ArrayList<>();
-    final ParameterDefn formatProp = new Parameter(FORMAT_PARAMETER, ParameterType.VARCHAR, false);
+    // While the format parameter is required, we mark it as optional. Else
+    // if the source defines optional parameters, they will still be ignored

Review Comment:
   if the format* defines optional parameters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102107289


##########
server/src/main/java/org/apache/druid/catalog/model/table/FormattedInputSourceDefn.java:
##########
@@ -95,7 +95,10 @@ protected List<ParameterDefn> addFormatParameters(
   )
   {
     final List<ParameterDefn> toAdd = new ArrayList<>();
-    final ParameterDefn formatProp = new Parameter(FORMAT_PARAMETER, ParameterType.VARCHAR, false);
+    // While the format parameter is required, we mark it as optional. Else
+    // if the source defines optional parameters, they will still be ignored
+    // as Calcite treats (optional, optional, required) as (required, required, required)

Review Comment:
   What do you mean by this? Do you mean that if any of the parameters are required then all are? If thats the case, then effectively, how can we have any optional parameters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102202501


##########
server/src/main/java/org/apache/druid/catalog/model/table/FormattedInputSourceDefn.java:
##########
@@ -95,7 +95,10 @@ protected List<ParameterDefn> addFormatParameters(
   )
   {
     final List<ParameterDefn> toAdd = new ArrayList<>();
-    final ParameterDefn formatProp = new Parameter(FORMAT_PARAMETER, ParameterType.VARCHAR, false);
+    // While the format parameter is required, we mark it as optional. Else
+    // if the source defines optional parameters, they will still be ignored

Review Comment:
   Actually, _input source_, as written. The input source parameters preceded the input format parameters. That's how we end up with the scenario in the comment: the input source has two optional parameters, but the format has a required one. That forces the input source parameters to also become required.
   
   This is, by the way, why that bug in `extern`, where I had the optional flag backward, actually worked. Calcite took my `(optional, optional, required)` and turned it into what the code originally had: `(required, required, required)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102201112


##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -238,6 +241,7 @@ protected InputSource convertSource(
       return jsonMapper.convertValue(jsonMap, inputSourceClass());
     }
     catch (Exception e) {
+      LOG.debug(e, "Invalid input source specification");

Review Comment:
   This is for debugging. Specifically, the case in which an input format requires a JSON or Guice thing which is not configured in the Coordinator. Only going to happen for developers, not for users.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekrb19 commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1101248487


##########
server/src/main/java/org/apache/druid/catalog/model/table/HttpInputSourceDefn.java:
##########
@@ -172,6 +172,14 @@ public void validate(ResolvedExternalTable table)
     super.validate(table);
   }
 
+  @Override
+  protected void auditInputSource(Map<String, Object> jsonMap)
+  {
+    // A partial table may not include the URI parameter, but it is
+    // needed to serialize the input source.
+    jsonMap.putIfAbsent(URIS_PARAMETER, "http://bogus.com");

Review Comment:
   Why do we need to populate a bogus value here? Is it used to bypass any validation checks?



##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseTableFunction.java:
##########
@@ -76,11 +85,11 @@ public List<ParameterDefn> parameters()
     return parameters;
   }
 
-  protected void requireSchema(String fnName, List<ColumnSpec> columns)
+  protected static void requireSchema(String fnName, List<ColumnSpec> columns)
   {
     if (columns == null) {
       throw new IAE(
-          "The %s table function requires an EXTEND clause with a schema",
+          "Function requires a schema: TABLE(%s(...)) (<col> <type>...)",

Review Comment:
   Neat



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1103370272


##########
server/src/main/java/org/apache/druid/catalog/model/table/HttpInputSourceDefn.java:
##########
@@ -172,6 +172,14 @@ public void validate(ResolvedExternalTable table)
     super.validate(table);
   }
 
+  @Override
+  protected void auditInputSource(Map<String, Object> jsonMap)
+  {
+    // A partial table may not include the URI parameter, but it is
+    // needed to serialize the input source.
+    jsonMap.putIfAbsent(URIS_PARAMETER, "http://bogus.com");

Review Comment:
   Everyone asks this. I'm sad that the comment doesn't explain it. You can have a partial input source definition: "with URIs to be named later." But, our input source class requires all the details. In order to validate the bits we _do_ have , we have to make up bits to make Druid happy.
   
   This is when validating a (partial) catalog definition. When the user uses this spec in a function, we do require all information be provided to create a "complete" table. We don't use bogus values at that time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102104860


##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -238,6 +241,7 @@ protected InputSource convertSource(
       return jsonMapper.convertValue(jsonMap, inputSourceClass());
     }
     catch (Exception e) {
+      LOG.debug(e, "Invalid input source specification");

Review Comment:
   Should this be error/warning log? debug isnt logged by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers merged pull request #13780: Refinements to input-source specific table functions

Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers merged PR #13780:
URL: https://github.com/apache/druid/pull/13780


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102107289


##########
server/src/main/java/org/apache/druid/catalog/model/table/FormattedInputSourceDefn.java:
##########
@@ -95,7 +95,10 @@ protected List<ParameterDefn> addFormatParameters(
   )
   {
     final List<ParameterDefn> toAdd = new ArrayList<>();
-    final ParameterDefn formatProp = new Parameter(FORMAT_PARAMETER, ParameterType.VARCHAR, false);
+    // While the format parameter is required, we mark it as optional. Else
+    // if the source defines optional parameters, they will still be ignored
+    // as Calcite treats (optional, optional, required) as (required, required, required)

Review Comment:
   What do you mean by this? Do you mean that if any of the parameters are required then all are? If thats the case, then effectively, how can we have any optional parameters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102107289


##########
server/src/main/java/org/apache/druid/catalog/model/table/FormattedInputSourceDefn.java:
##########
@@ -95,7 +95,10 @@ protected List<ParameterDefn> addFormatParameters(
   )
   {
     final List<ParameterDefn> toAdd = new ArrayList<>();
-    final ParameterDefn formatProp = new Parameter(FORMAT_PARAMETER, ParameterType.VARCHAR, false);
+    // While the format parameter is required, we mark it as optional. Else
+    // if the source defines optional parameters, they will still be ignored
+    // as Calcite treats (optional, optional, required) as (required, required, required)

Review Comment:
   What do you mean by this? Do you mean that if any of the parameters are required then all are?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zachjsh commented on a diff in pull request #13780: Refinements to input-source specific table functions

Posted by "zachjsh (via GitHub)" <gi...@apache.org>.
zachjsh commented on code in PR #13780:
URL: https://github.com/apache/druid/pull/13780#discussion_r1102104860


##########
server/src/main/java/org/apache/druid/catalog/model/table/BaseInputSourceDefn.java:
##########
@@ -238,6 +241,7 @@ protected InputSource convertSource(
       return jsonMapper.convertValue(jsonMap, inputSourceClass());
     }
     catch (Exception e) {
+      LOG.debug(e, "Invalid input source specification");

Review Comment:
   Should this be error log? debug isnt logged by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org