You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2026/01/29 19:12:09 UTC
[PR] SQL: Use specialized virtual columns for expression filters. (druid)
gianm opened a new pull request, #18965:
URL: https://github.com/apache/druid/pull/18965
This patch adjusts planning for expression filters to use specialized virtual columns when they exist. This allows them to take advantage of optimizations, such as the ones that are available for JSON_VALUE, even when the overall expression is complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] SQL: Use specialized virtual columns for expression filters. (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18965:
URL: https://github.com/apache/druid/pull/18965#discussion_r2743242191
##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/Expressions.java:
##########
@@ -831,25 +831,43 @@ private static DimFilter toSimpleLeafFilter(
/**
* Translates to an "expression" type leaf filter. Used as a fallback if we can't use a simple leaf filter.
+ * If the expression contains specialized nodes (e.g. JSON_VALUE), a virtual column is created to benefit
+ * from specialization. Otherwise, the expression is inlined directly.
*/
@Nullable
private static DimFilter toExpressionLeafFilter(
final PlannerContext plannerContext,
final RowSignature rowSignature,
+ @Nullable final VirtualColumnRegistry virtualColumnRegistry,
final RexNode rexNode
)
{
final DruidExpression druidExpression = toDruidExpression(plannerContext, rowSignature, rexNode);
- if (druidExpression != null) {
+ if (druidExpression == null) {
+ return null;
+ }
+
+ if (virtualColumnRegistry != null && druidExpression.containsSpecializedNodes()) {
+ // Use a virtual column so we can take advantage of specialization.
+ final String virtualColumnName = virtualColumnRegistry.getOrCreateVirtualColumnForExpression(
+ druidExpression,
+ ColumnType.LONG
+ );
+ final String vcExpression = "\"" + virtualColumnName + "\"";
Review Comment:
first impression is it looks a bit weird that the expression filter is just an identifier expr a pointer to a vc, but internally that is basically what happens anyway so it seems fine to me after getting past the strangeness, and this i think would also allow expr re-use between filters and other things so :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] SQL: Use specialized virtual columns for expression filters. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #18965:
URL: https://github.com/apache/druid/pull/18965#discussion_r2748096316
##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/Expressions.java:
##########
@@ -831,25 +831,43 @@ private static DimFilter toSimpleLeafFilter(
/**
* Translates to an "expression" type leaf filter. Used as a fallback if we can't use a simple leaf filter.
+ * If the expression contains specialized nodes (e.g. JSON_VALUE), a virtual column is created to benefit
+ * from specialization. Otherwise, the expression is inlined directly.
*/
@Nullable
private static DimFilter toExpressionLeafFilter(
final PlannerContext plannerContext,
final RowSignature rowSignature,
+ @Nullable final VirtualColumnRegistry virtualColumnRegistry,
final RexNode rexNode
)
{
final DruidExpression druidExpression = toDruidExpression(plannerContext, rowSignature, rexNode);
- if (druidExpression != null) {
+ if (druidExpression == null) {
+ return null;
+ }
+
+ if (virtualColumnRegistry != null && druidExpression.containsSpecializedNodes()) {
+ // Use a virtual column so we can take advantage of specialization.
+ final String virtualColumnName = virtualColumnRegistry.getOrCreateVirtualColumnForExpression(
+ druidExpression,
+ ColumnType.LONG
+ );
+ final String vcExpression = "\"" + virtualColumnName + "\"";
Review Comment:
I did think about using an `equals` filter in this case (like, `equals` with `1L` value) but decided against it because of the fancy coercing that the `expression` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] SQL: Use specialized virtual columns for expression filters. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #18965:
URL: https://github.com/apache/druid/pull/18965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org