You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Vivek Ratan (JIRA)" <ji...@apache.org> on 2008/07/31 07:24:31 UTC
[jira] Created: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Hadoop Core should support source filesfor multiple schedulers
--------------------------------------------------------------
Key: HADOOP-3876
URL: https://issues.apache.org/jira/browse/HADOOP-3876
Project: Hadoop Core
Issue Type: New Feature
Reporter: Vivek Ratan
Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "David Litster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655416#action_12655416 ]
David Litster commented on HADOOP-3876:
---------------------------------------
Sorry to add my $0.02 so late in the process, but since you already use Torque and Condor to actually spawn the Hadoop Clusters and start jobs, have you considered adding functionality to HOD to allow external widely-used schedulers (such as the open-source Maui, Moab, PBSpro, LSF, etc. ) to control the scheduling of HOD clusters and jobs via the above-mentioned API (or the command-line or web service APIs)? This would allow sites that already have an existing scheduler and want to add the ability to run Hadoop jobs to be able to do so while taking advantage of their existing infrastructure in terms of users, SLAs, priorities, accounts, groups, etc.
Thanks for all the effort!
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618616#action_12618616 ]
Vivek Ratan commented on HADOOP-3876:
-------------------------------------
My guess is, we're going to have some 'core' schedulers (which seem appropriate to belong within the core Hadoop code), and some that are better suited as contrib projects. We should probably place core schedulers under _src/mapred/schedulers_. So, for example, the 3445 scheduler would go under _src/mapred/schedulers/3445_ and the 3746 scheduler under _src/mapred/schedulers/3746_ (replace '3445' and '3746' with more appropriate names, if you wish). Others may go in here as well, if its' felt that they're likely to be deployed in many scenarios, or for whatever other reason. Non-core schedulers can probably go under contrib.
Another option is to place core schedulers under _src/core/schedulers_, if we want these to be more than just MR schedulers, but folks may not write non-MR schedulers for a while. I prefer keeping stuff under _mapred_.
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620592#action_12620592 ]
Tom White commented on HADOOP-3876:
-----------------------------------
+1 on subpackages (either in src/mapred or src/contrib).
Also, I think that we need to split out the MapReduce daemons into a server package (HADOOP-3916) - which we won't publish javadoc for - before we can do HADOOP-3822 properly. This argues for committing the fair scheduler HADOOP-3746 as a contrib package (as it currently stands) - this can be done now, or into core after HADOOP-3916 and HADOOP-3822 have been done.
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618613#action_12618613 ]
Vivek Ratan commented on HADOOP-3876:
-------------------------------------
Matei [brought up this issue|https://issues.apache.org/jira/browse/HADOOP-3746?focusedCommentId=12618454#action_12618454].
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655487#action_12655487 ]
Hemanth Yamijala commented on HADOOP-3876:
------------------------------------------
David,
Some clarifications:
- We do not support Condor in HOD, only Torque
- HOD serves its purpose well as a good provisioning system for Hadoop, and is likely to live on to assist test environments where a large cluster can be shared by multiple developers / testers each having their own version of Hadoop to deploy and test.
- I think it it unlikely for HOD to exist as a scheduling solution for Hadoop, built on top of the resource managers you've mentioned in your comment. That functionality is added by scheduling solutions being built into Hadoop itself - for e.g. the Capacity Scheduler (HADOOP-3445) and the Fair Scheduler (HADOOP-3746).
- We did consider adding support for other resource managers in HOD, precisely for reasons you state - it will have an easier entry into places which are using other open source resource manager / scheduling solutions. However, given the shift in direction from HOD, this consideration did not proceed further.
- The API in HOD did plan for accomodating other schedulers - you could look at src/contrib/hod/hodlib/Hod/nodePool.py. It is likely that the abstraction may not be fully suitable, but still this would be the place to start if we want to plugin support for other schedulers.
Hope that answers some of your questions.
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3876) Hadoop Core should support source
filesfor multiple schedulers
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618899#action_12618899 ]
Matei Zaharia commented on HADOOP-3876:
---------------------------------------
Just a note, if we do subpackages, we will need a semi-public scheduler API (HADOOP-3822), because the default visibility in Java doesn't apply in subpackages. On the other hand, I think subpackages is definitely the way to go to make this scalable and clean.
> Hadoop Core should support source filesfor multiple schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-3876
> URL: https://issues.apache.org/jira/browse/HADOOP-3876
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Vivek Ratan
>
> Besides the default JT scheduling algorithm, there is work going on with at least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it easier to plug in new schedulers into the JT. Where do we place the source files for various schedulers so that it's easy for users to choose their scheduler of choice during deployment, and easy for developers to add in more schedulers into the framework (without inundating it).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.