[GitHub] [druid] adarshsanjeev commented on issue #13824: Aggregations on __time column do not work as expected

["adarshsanjeev (via GitHub)" <gi...@apache.org>]

adarshsanjeev commented on issue #13824:
URL: https://github.com/apache/druid/issues/13824#issuecomment-1438347030

   The cause of the problem seems to be present in StringLastAggregatorFactory, which is used in case of EARLIEST and LATEST. In case the time column is not known, we default to ColumnHolder.TIME_COLUMN_NAME ("__time"). This is under the assumption that a time column should be present and works for non MSQ queries. For some MSQ queries which read from an external source, the __time column is present in the output, but during aggregation, might be referred to by a temporary name or virtual column.
   
   An ideal solution would be to handle reading from aliased columns directly. This would help for queries like
   ```
     TIME_PARSE("timestamp") AS "__time",
     LATEST_BY("comment", "__time", 1024),
   ```
   which do not work currently.
   
   An alternate solution could be to handle EARLIEST and LATEST as a special case for now. We could change the implicit reference to the __time column. MSQTaskQueryMaker has the necessary mappings to know what is mapped to the __time column in the output. ColumnMappings contains the mapping of __time to a the intermediate column, (MSQ sets CTX_TIME_COLUMN_NAME to this in its query context) and dimensions contains the mappings of virtual columns. Changing this reference from __time to the column that is mapped to it in the final output produces the expected output of latest in the above query with LATEST. This might need some additional changes to support compaction, but it could be able to handle this case if the reference to the column is changed to the __time column during the process.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org