You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2022/08/31 14:28:18 UTC
[GitHub] [drill] jnturton opened a new pull request, #2636: DRILL-8290: Short circuit recursive file listings for LIMIT 0 queries.
jnturton opened a new pull request, #2636:
URL: https://github.com/apache/drill/pull/2636
# [DRILL-8290](https://issues.apache.org/jira/browse/DRILL-8290): Short circuit recursive file listings for LIMIT 0 queries
## Description
The existing LIMIT 0 query optimisations do not prevent a query run against the top of a deep DFS directory tree from recursively listing FileStatuses for everything within it using a pool of worker threads. This PR adds a new optimisation whereby such queries will recurse into the directory tree on a single thread that returns as soon as any single FileStatus has been obtained.
## Documentation
Mention in the docs on LIMIT 0 optimisations.
## Testing
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton merged pull request #2636: DRILL-8290: Early exit from recursive file listing for LIMIT 0 queries
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton merged PR #2636:
URL: https://github.com/apache/drill/pull/2636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on pull request #2636: DRILL-8290: Short cut recursive file listings for LIMIT 0 queries.
Posted by GitBox <gi...@apache.org>.
jnturton commented on PR #2636:
URL: https://github.com/apache/drill/pull/2636#issuecomment-1398472306
> For such queries the same QueryComputationHints will be used for both inputs, so it will cause incorrect results.
@vvysotskyi the idea here was that only a LIMIT 0 on the _root_ SELECT is detected, in which case the single file optimisation can be done on _all_ inputs so a single flag is sufficient.
However, I'm trying to implement a better approach that optimises LIMIT 0s at any level. Since files are first listed very early, during validation (so even before partition pruning 🙁) no RelNode trees are available and the detection will have to be done on the SqlNode tree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on pull request #2636: DRILL-8290: Short cut recursive file listings for LIMIT 0 queries.
Posted by GitBox <gi...@apache.org>.
jnturton commented on PR #2636:
URL: https://github.com/apache/drill/pull/2636#issuecomment-1234361083
@vvysotskyi I did spot one [other recursive file listing](https://github.com/jnturton/drill/blob/65fb7ddc144ecae5330c9325af63010748f74cdf/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java#L376) that could possibly use the short cut in this PR if I propagate a `limit0` flag down to it.
It appears to be invoked only if there are Parquet files present at the top level of the queried path which I don't think should be too common for big datasets since data files are generally only present at the leaves of the directory tree. So I thought I'd ask if you think it's worth trying to implement the single file short cut here too, or we just leave it alone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vvysotskyi commented on a diff in pull request #2636: DRILL-8290: Early exit from recursive file listing for LIMIT 0 queries
Posted by "vvysotskyi (via GitHub)" <gi...@apache.org>.
vvysotskyi commented on code in PR #2636:
URL: https://github.com/apache/drill/pull/2636#discussion_r1096586279
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java:
##########
@@ -645,10 +646,18 @@ public Void visitOp(PhysicalOperator op, Collection<PhysicalOperator> collection
}
return null;
}
-
}
protected Pair<SqlNode, RelDataType> validateNode(SqlNode sqlNode) throws ValidationException, RelConversionException, ForemanSetupException {
+ if (context.getOptions().getOption(ExecConstants.FILE_LISTING_LIMIT0_OPT)) {
+ // Check for a LIMIT 0 in the root portion of the query before validation
+ // because validation of the query's FROM clauses will already trigger
+ // the recursive listing files to which FILE_LISTING_LIMIT0_OPT is meant
+ // to apply.
+ boolean rootSelectLimit0 = FindLimit0SqlVisitor.containsLimit0(sqlNode);
+ context.getPlannerSettings().setRootSelectLimit0(rootSelectLimit0);
Review Comment:
Perhaps with the query option would be simpler to pass the value rather than extend the planner setting interface and pass it through. And schema config already has access to query options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on a diff in pull request #2636: DRILL-8290: Early exit from recursive file listing for LIMIT 0 queries
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on code in PR #2636:
URL: https://github.com/apache/drill/pull/2636#discussion_r1099737668
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java:
##########
@@ -645,10 +646,18 @@ public Void visitOp(PhysicalOperator op, Collection<PhysicalOperator> collection
}
return null;
}
-
}
protected Pair<SqlNode, RelDataType> validateNode(SqlNode sqlNode) throws ValidationException, RelConversionException, ForemanSetupException {
+ if (context.getOptions().getOption(ExecConstants.FILE_LISTING_LIMIT0_OPT)) {
+ // Check for a LIMIT 0 in the root portion of the query before validation
+ // because validation of the query's FROM clauses will already trigger
+ // the recursive listing files to which FILE_LISTING_LIMIT0_OPT is meant
+ // to apply.
+ boolean rootSelectLimit0 = FindLimit0SqlVisitor.containsLimit0(sqlNode);
+ context.getPlannerSettings().setRootSelectLimit0(rootSelectLimit0);
Review Comment:
@vvysotskyi I've made use of a query option, how do things look now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org