[GitHub] [druid] maytasm commented on issue #5721: Ingest: need a way to determine when new data is queryable

[GitBox <gi...@apache.org>]

maytasm commented on issue #5721:
URL: https://github.com/apache/druid/issues/5721#issuecomment-618688438


   loadstatus API reads segments from SqlSegmentsMetadataManager of the Coordinator which caches segments in memory and periodically updates them. Hence, there can be a race condition as this API implementation compares segments metadata from cache with segments in historicals. Particularly, when number of segment changes (new ingestion) after the initial load of the datasource, the cache still only contains the metadata of old segments. I notice this too when trying to verify integration tests. The workaround I did (which may not be applicable to real life production) is to verify the number of segments matches expected from metadata/datasources/{DATASOURCE}/segments API before calling this method (since, that would wait until the cache is updated with expected data).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org