You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2024/06/07 22:09:39 UTC

[PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

clintropolis opened a new pull request, #16577:
URL: https://github.com/apache/druid/pull/16577

   ### Description
   Fixes an issue that occurs after #16366 where expressions with single input dictionary encoded columns that are not strings but have a string output incorrectly use the string dictionary encoded vector selector.
   
   The fix for now is to make setting the capabilities of the virtual column to be dictionary encoded only if the input type is string. In the future if we make dictionary encoded selectors that are not coupled with handling strings then we could remove this constraint, but until then, this is the safest option to avoid incorrectly using string dimension selectors.
   
   While here also noticed that `COMPLEX<json>` was needlessly reporting itself as dictionary encoded, which isn't quite true. While the nested field columns are dictionary encoded, the json values themselves are not, so functions like `TO_JSON_STRING` could also run into this problem.
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [x] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on PR #16577:
URL: https://github.com/apache/druid/pull/16577#issuecomment-2611993686

   I think this caused a regression.
   I have a search query that use to run successfully but now failed with
   ```
   Cannot invoke "org.apache.druid.segment.index.semantic.DictionaryEncodedStringValueIndex.getCardinality()" because "bitmapIndex" is null
   java.lang.NullPointerException
   ```
   This is the query:
   ```
   {"queryType":"search","dataSource":"wikipedia","granularity":"all","intervals":["2000-11-22/2024-12-21"],"virtualColumns":[{"name":"spark_cluster_group","type":"expression","expression":"case_searched(like(cityName,'C%'), 'foo', like(cityName, 'B%'), 'bar', 'other')","outputType":"STRING"}],"searchDimensions":["spark_cluster_group"],"query":{"type":"insensitive_contains","value":""},"sort":{"type":"lexicographic"},"limit":100}
   ```
   This query can be reproduced with the wikipedia datasource


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631786512


##########
processing/src/main/java/org/apache/druid/segment/serde/NestedCommonFormatColumnPartSerde.java:
##########
@@ -303,10 +303,6 @@ public void read(ByteBuffer buffer, ColumnBuilder builder, ColumnConfig columnCo
           bitmapSerdeFactory,
           byteOrder
       );
-      ColumnCapabilitiesImpl capabilitiesBuilder = builder.getCapabilitiesBuilder();

Review Comment:
   ah, put in the description:
   
   >While here also noticed that COMPLEX<json> was needlessly reporting itself as dictionary encoded, which isn't quite true. While the nested field columns are dictionary encoded, the json values themselves are not, so functions like TO_JSON_STRING could also run into this problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631790888


##########
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlan.java:
##########
@@ -277,6 +277,9 @@ public ColumnCapabilities inferColumnCapabilities(@Nullable ColumnType outputTyp
             // until query time
             return ColumnCapabilitiesImpl.copyOf(underlyingCapabilities)
                                          .setType(ColumnType.STRING)
+                                         // this is sad, but currently dictionary encodedness is tied to string
+                                         // selectors and sad stuff happens if the input type isn't string
+                                         .setDictionaryEncoded(underlyingCapabilities.is(ValueType.STRING))

Review Comment:
   actually, we were explicitly marking hasBitmapIndexes as false, though #15585 allows expressions to use them in many cases, so it probably should be allowed to pass through.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631789920


##########
processing/src/main/java/org/apache/druid/segment/serde/NestedCommonFormatColumnPartSerde.java:
##########
@@ -303,10 +303,6 @@ public void read(ByteBuffer buffer, ColumnBuilder builder, ColumnConfig columnCo
           bitmapSerdeFactory,
           byteOrder
       );
-      ColumnCapabilitiesImpl capabilitiesBuilder = builder.getCapabilitiesBuilder();

Review Comment:
   ahh, I missed that the `capabilitiesBuilder` is from `builder` itself. OK, this change makes sense then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631790888


##########
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlan.java:
##########
@@ -277,6 +277,9 @@ public ColumnCapabilities inferColumnCapabilities(@Nullable ColumnType outputTyp
             // until query time
             return ColumnCapabilitiesImpl.copyOf(underlyingCapabilities)
                                          .setType(ColumnType.STRING)
+                                         // this is sad, but currently dictionary encodedness is tied to string
+                                         // selectors and sad stuff happens if the input type isn't string
+                                         .setDictionaryEncoded(underlyingCapabilities.is(ValueType.STRING))

Review Comment:
   actually, we were explicitly marking hasBitmapIndexes as false, though #15585 allows expressions to use them in many cases, so it probably should be allowed to pass through. `hasBitmapIndexes` and `hasSpatialIndexes` should probably be removed from `ColumnCapabilities`, and the write side stuff that currently uses it be moved to `ColumnFormat`, though not going to do that on this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631786003


##########
processing/src/main/java/org/apache/druid/segment/serde/NestedCommonFormatColumnPartSerde.java:
##########
@@ -303,10 +303,6 @@ public void read(ByteBuffer buffer, ColumnBuilder builder, ColumnConfig columnCo
           bitmapSerdeFactory,
           byteOrder
       );
-      ColumnCapabilitiesImpl capabilitiesBuilder = builder.getCapabilitiesBuilder();

Review Comment:
   this is just removing unused code?



##########
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlan.java:
##########
@@ -277,6 +277,9 @@ public ColumnCapabilities inferColumnCapabilities(@Nullable ColumnType outputTyp
             // until query time
             return ColumnCapabilitiesImpl.copyOf(underlyingCapabilities)
                                          .setType(ColumnType.STRING)
+                                         // this is sad, but currently dictionary encodedness is tied to string
+                                         // selectors and sad stuff happens if the input type isn't string
+                                         .setDictionaryEncoded(underlyingCapabilities.is(ValueType.STRING))

Review Comment:
   shouldn't this be `underlyingCapabilities.isTrue() && underlyingCapabilities.is(ValueType.STRING)`?



##########
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlan.java:
##########
@@ -277,6 +277,9 @@ public ColumnCapabilities inferColumnCapabilities(@Nullable ColumnType outputTyp
             // until query time
             return ColumnCapabilitiesImpl.copyOf(underlyingCapabilities)
                                          .setType(ColumnType.STRING)
+                                         // this is sad, but currently dictionary encodedness is tied to string
+                                         // selectors and sad stuff happens if the input type isn't string
+                                         .setDictionaryEncoded(underlyingCapabilities.is(ValueType.STRING))

Review Comment:
   also, imo `copyOf` is not the right thing to use here, logically speaking. I don't think we actually want to carry through all caps from the underlying selector, just a few specific ones.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631787651


##########
processing/src/main/java/org/apache/druid/segment/virtual/ExpressionPlan.java:
##########
@@ -277,6 +277,9 @@ public ColumnCapabilities inferColumnCapabilities(@Nullable ColumnType outputTyp
             // until query time
             return ColumnCapabilitiesImpl.copyOf(underlyingCapabilities)
                                          .setType(ColumnType.STRING)
+                                         // this is sad, but currently dictionary encodedness is tied to string
+                                         // selectors and sad stuff happens if the input type isn't string
+                                         .setDictionaryEncoded(underlyingCapabilities.is(ValueType.STRING))

Review Comment:
   >shouldn't this be underlyingCapabilities.isTrue() && underlyingCapabilities.is(ValueType.STRING)?
   
   oops yes.
   
   >also, imo copyOf is not the right thing to use here, logically speaking. I don't think we actually want to carry through all caps from the underlying selector, just a few specific ones.
   
   Ah yea that is reasonable i suppose, i think originally more were preserved than not so i went with the copy. I guess `hasBitmapIndexes` isn't widely used on query side anymore since stuff can just ask `ColumnIndexSupplier` for an index and it will return if it has it, though i suppose it should still be preserved just in case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #16577:
URL: https://github.com/apache/druid/pull/16577


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix expression column capabilities to not report dictionary encoded unless input is string (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16577:
URL: https://github.com/apache/druid/pull/16577#discussion_r1631786512


##########
processing/src/main/java/org/apache/druid/segment/serde/NestedCommonFormatColumnPartSerde.java:
##########
@@ -303,10 +303,6 @@ public void read(ByteBuffer buffer, ColumnBuilder builder, ColumnConfig columnCo
           bitmapSerdeFactory,
           byteOrder
       );
-      ColumnCapabilitiesImpl capabilitiesBuilder = builder.getCapabilitiesBuilder();

Review Comment:
   ah, i had put in the description:
   
   >While here also noticed that COMPLEX<json> was needlessly reporting itself as dictionary encoded, which isn't quite true. While the nested field columns are dictionary encoded, the json values themselves are not, so functions like TO_JSON_STRING could also run into this problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org