[jira] Commented: (HADOOP-3930) Decide how to integrate scheduler info into CLI and job tracker web page

["Vivek Ratan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621865#action_12621865 ] 

Vivek Ratan commented on HADOOP-3930:
-------------------------------------

I think we need to first decide whether queues are explicit in this API or not. The problem with making queues explicit in the API is that every scheduler will have to support one,  or at least a default one. But that's not so bad, IMO. 

getSchedulingInfo() should really return key-value pairs for queues, not for jobs. In the HADOOP-3445 scheduler, for example, we need to display scheduling information associated with a queue - its capacity (both 'guaranteed' and 'allocated'), how many unique users have submitted jobs, how many tasks are running, how many are waiting. etc. This information is per queue, and doesn't make sense per job. I'd much rather have getSchedulingInfo() take in a queue name as a parameter, if we make queues explicit. In fact, I don't see what kind of scheduling information you'd associate with a job. Matei, do you have examples of what getSchedulingInfo would return for jobs? 

Similarly, getJobComparator() makes more sense when applied to a queue. In 3445, jobs are ordered per queue, and there is no global ordering. Furthermore, doesn't it make more sense to get a sorted collection of jobs, per queue, back from the scheduler, rather than a Comparator? Or are you imagining the UI and CLI to maintain a list of jobs all the time and then apply the comparator periodically? 



> Decide how to integrate scheduler info into CLI and job tracker web page
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-3930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3930
>             Project: Hadoop Core
>          Issue Type: Improvement
>            Reporter: Matei Zaharia
>            Priority: Minor
>
> We need a way for job schedulers such as HADOOP-3445 and HADOOP-3476 to provide info to display on the JobTracker web interface and in the CLI. The main things needed seem to be:
> * A way for schedulers to provide info to show in a column on the web UI and in the CLI - something as simple as a single string, or a map<string, int> for multiple parameters.
> * Some sorting order for jobs - maybe a method to sort a list of jobs.
> Let's figure out what the best way to do this is and implement it in the existing schedulers.
> My first-order proposal at an API: Augment the TaskScheduler with
> * public Map<String, String> getSchedulingInfo(JobInProgress job) -- returns key-value pairs which are displayed in columns on the web UI or the CLI.
> * public Comparator<JobInProgress> getJobComparator() -- returns a comparator that can be used to determine the order in which jobs will be run, for sorting the jobs in the CLI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.