You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "paul-rogers (via GitHub)" <gi...@apache.org> on 2023/02/08 00:05:57 UTC
[GitHub] [druid] paul-rogers opened a new issue, #13770: Delay reporting MSQ ingest success until segments are loaded
paul-rogers opened a new issue, #13770:
URL: https://github.com/apache/druid/issues/13770
### Affected Version
26.0.0-SNAPSHOT
### Description
I am creating a Jupyter notebook to illustrate how to us the new Druid catalog. As part of that task, I submit an MSQ ingestion task, wait for the Overlord to report task completion, then query the table. Each ingestion uses `REPLACE` and usually creates a new datasource.
When running queries, I occasionally (about 20% of the time) get an error saying that there is no such table. Yet, if I wait a few seconds, and try again, the query succeeds. The reason is clear: MSQ reported success as soon as ingestion is complete. It takes a while for the new segments to be loaded onto my one historical node. During that time, the Broker knows nothing about the new table.
To be very specific:
* No segments for the target table exist.
* Call `/sql/task` to submit an MSQ `REPLACE` query.
* Poll Overlord waiting for the task to be marked as completed.
* Immediately issue a `/sql` query against that same table.
This creates a race condition. Druid reports that the ingest is done, but it is not really done. The client has to be smart enough to know that the resulting query error is due to a race condition, not to one of possibly many other problems. This puts the burden on the client. Or, in my case, I have to add extra verbiage that says "if this query fails, wait a while and try again", which doesn't scream "easy to use."
The ask is for MSQ to wait for segments to be loaded before declaring completion. That way, a client that waits for MSQ task completion can be assured that, when the task is complete, the table is actually ready to be queried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Delay reporting MSQ ingest success until segments are loaded (druid)
Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla closed issue #13770: Delay reporting MSQ ingest success until segments are loaded
URL: https://github.com/apache/druid/issues/13770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Delay reporting MSQ ingest success until segments are loaded (druid)
Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on issue #13770:
URL: https://github.com/apache/druid/issues/13770#issuecomment-1715054711
I think https://github.com/apache/druid/pull/14322 completes all the requirements mentioned in the PR. cc: @adarshsanjeev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org