You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2025/07/30 00:04:40 UTC

[I] Realtime scans from MSQ cannot reliably read complex types (druid)

gianm opened a new issue, #18340:
URL: https://github.com/apache/druid/issues/18340

   When `ScanQueryFrameProcessor` delegates work to a realtime server, it sends a `scan` query and writes the response into a frame. Their serialization when selected in a `scan` query is done by Jackson, and it is common for such objects to have "one-way" serialization (they can be serialized using Jackson, but cannot be deserialized).
   
   For aggregating queries, typically these objects are deserialized by the associated aggregator's `AggregatorFactory#deserialize` method rather than by Jackson. This is problematic for `scan` queries, because they don't use `AggregatorFactory`. This leads to errors when attempting to write the resulting objects to a frame, because they aren't the expected type.
   
   To fix it, we need to serialize and deserialize these complex objects in a coherent way. There are a couple options that come to mind,
   
   - We could add a new parameter to `scan` that causes it to serialize complex objects using `TypeStrategy` rather than allowing Jackson to handle it. MSQ realtime delegation would use this parameter. Then, when reading, we use the same `TypeStrategy` to deserialize the object.
   - We could rewrite `scan` queries in MSQ's realtime delegation code to wrap all complex types in `complex_encode_base64`, which would accomplish more or less the same thing- it would use the `TypeStrategy` to serialize the object.
   
   This bug affects all MSQ queries that use realtime data and delegate `scan` involving complex types. An example query would be like:
   
   ```sql
   SELECT "hllSketch"
   FROM tbl
   WHERE "foo" = 'bar'
   ```
   
   Queries that involve aggregations on complex types wouldn't be affected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-3737404989

   I will move this out from 36.0.0 (into 37.0.0) for now - it doesn't seem like a new issue which should block the release.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-3160361705

   Note that this bug affects scan pushdown only. Aggregation pushdown, which is more commonly what happens with complex types, is not affected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-4203941597

   Yes let's move it to 38.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-5353977415

   I'd like to work on this issue. Could you please assign it to me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "cecemei (via GitHub)" <gi...@apache.org>.
cecemei commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-4201164823

   @gianm  whats the status for this issue? should we move this to 38?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-5187403333

   Moving this milestone from 38.0.0 to 39.0.0 as there has not been a PR yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Realtime scans from MSQ cannot reliably read complex types (druid)

Posted by "nozjkoitop (via GitHub)" <gi...@apache.org>.
nozjkoitop commented on issue #18340:
URL: https://github.com/apache/druid/issues/18340#issuecomment-4681830282

   Hey @gianm
   
   This looks a lot like the issue I was seeing before. You might want to take a look at #16607 (later refreshed as #18859), it fixes a very similar MSQ realtime path problem with complex types/HLL sketches
   
   Not sure it covers every case here, but it seems closely related, it might be worth reopening #18859 and merging the latest changes rather than starting from scratch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org