[GitHub] [spark] Ngone51 commented on pull request #32136: [SPARK-35022][CORE] Task Scheduling Plugin in Spark

[GitBox <gi...@apache.org>]

Ngone51 commented on pull request #32136:
URL: https://github.com/apache/spark/pull/32136#issuecomment-844775582


   Thanks for the ping @xuanyuanking 
   
   > does it fit into stage level scheduling
   
   This sounds feasible to me. We can treat the state store as a resource for the streaming task. And since the `StateStoreProviderId` is shared between batches, tasks between batches must be assigned the same state store as long as they require the same `StateStoreProviderId` (which is guaranteed by the stage level scheduling mechanism). Here's pseudo code may look like:
   
   ```scala
   case class StateStoreRDD {
   
    ...
    this. withResources(new ResourceProfile().add(StateStoreProviderId))
    ...
   }
   ```
   
   On the other side, driver should be able to update `ExecutorData.resourcesInfo` when `StateStoreCoordinatorRef` receives the active state store instance register so that the executor would contain the state store resource.
   
   One thing we need to pay attention to is that: there're might be no available executors for the specific `StateStoreProviderId` due to executor lost or 1st batch (where state store hasn't established
   ) which could leads the scheduling hang. Thus, I'm thinking of making the state store as an "optional" resource.
   
   While stage level scheduling solving the "must run at a particular executor" problem, the problem of unenvely distribution of the first batch still exits. I don't have a good idea yet. But I think we can add hack code in scheduler anyway (e.g., we can add the strategy as you added in the the #32422) as long as we know its the 1st batch.
   
   Thoughts?
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org