[GitHub] [pinot] t0mpere commented on issue #11282: Minion Batch ingestion scheduling bottleneck

["t0mpere (via GitHub)" <gi...@apache.org>]

t0mpere commented on issue #11282:
URL: https://github.com/apache/pinot/issues/11282#issuecomment-1670431417

   Ok so these are the logs from a job scheduling. As you can see the task generation is very quick and the scheduling seems to be the bottleneck. 19 seconds passed between generation and the response. Let me know if you need any more info.
   ```
   2023-08-04 16:50:00.118 BST Trying to create tasks of type: SegmentGenerationAndPushTask, table: TABLE
   2023-08-04 16:50:00.434 BST Submitting ad-hoc task for task type: SegmentGenerationAndPushTask with task configs: [...]
   2023-08-04 16:50:00.452 BST Submitting parent task: Task_SegmentGenerationAndPushTask_TABLE_c826881f-a0c5-48d0-bb58-8a43b3037b60 of type: SegmentGenerationAndPushTask with 1 child task configs
   2023-08-04 16:50:00.456 BST Add job configuration TaskQueue_SegmentGenerationAndPushTask_Task_SegmentGenerationAndPushTask_TABLE_c826881f-a0c5-48d0-bb58-8a43b3037b60
   2023-08-04 16:50:19.144 BST Handled request from 10.33.98.12 POST http://prod.host:80/tasks/execute, content-type application/json status code 200 OK
   2023-08-04 16:50:19.376 BST Trying to create tasks of type: SegmentGenerationAndPushTask, table: TABLE
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org