[GitHub] [incubator-druid] VladimirIordanov opened a new issue #7583: [Proposal] segmentMetadata query returns full list of dimensions

[GitBox <gi...@apache.org>]

VladimirIordanov opened a new issue #7583: [Proposal] segmentMetadata query returns full list of dimensions
URL: https://github.com/apache/incubator-druid/issues/7583
 
 
   ### Motivation
   
   Currently segmentMetadata query doesn't return dimensions which have no values (cardinality=0) for the time range queried. Some tools (i.e. Imply Pivot) use this type of query to fetch metadata and then validate all user requests against it. When these tools got in such situation with 'non-existing' dimension they treat the request as invalid. Although Druid Broker can process JSON request with such dimension correctly and return an expected result. Even more that these dimensions are displayed in timeline in coordinator views. So there is even inconsistency between it and segmentMetadata query.
   Situation got worse after introducing SQL to Druid. For now Druid fails when these dimensions are requested in SQL expression. Example of the error response:
   {
       "error": "Unknown exception",
       "errorMessage": "org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 18: Column 'xxx' not found in any table",
       "errorClass": "org.apache.calcite.tools.ValidationException",
       "host": null
   }
   It looks like Apache Calcite tries to follow the described workflow and match request to metadata and receives the same trouble.
   
   ### Proposed changes
   
   Change segmentMetadata query to output all dimensions even they do not have data in the requested time range.
   
   ### Rationale
   
   It is natural for data storages to have such dimensions with occasional data. The metadata of an data source should be the same for any time range even there are such dimensions in it. Behavior of Druid should be consistent across all its tools and do not change on any cases. Druid core should be consistent and SQL queries should not fail an error for requests which JSON equivalent works as expected.
   
   ### Operational impact
   
   No operational impact.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org