You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by olegz <gi...@git.apache.org> on 2014/09/17 05:05:41 UTC
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
GitHub user olegz opened a pull request:
https://github.com/apache/spark/pull/2422
SPARK-3561 - Pluggable strategy to facilitate access to native Hadoop/YARN resources
Added HadoopExecutionContext trait and its default implementation DefaultHadoopExecutionContext
Modified SparkContext to instantiate and delegate to the instance of HadoopExecutionContext where appropriate
Changed HadoopExecutionContext to JobExecutionContext
Changed DefaultHadoopExecutionContext to DefaultExecutionContext
Name changes are due to the fact that when Spark executes outside of Hadoop having Hadoop in the name woudl be confusing
Added initial documentation and tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/olegz/spark-1 SPARK-HADOOP
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2422.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2422
----
commit 15cd9b9110026b82a191ceb4d32f38230eec3b47
Author: Oleg Zhurakousky <ol...@suitcase.io>
Date: 2014-09-16T15:26:48Z
Initial commit to provide pluggable strategy to facilitate access to native Hadoop resources
Added HadoopExecutionContext trait and its default implementation DefaultHadoopExecutionContext
Modified SparkContext to instantiate and delegate to the instance of HadoopExecutionContext where appropriate
Changed HadoopExecutionContext to JobExecutionContext
Changed DefaultHadoopExecutionContext to DefaultExecutionContext
Name changes are due to the fact that when Spark executes outside of Hadoop having Hadoop in the name woudl be confusing
Added initial documentation and tests
polished scaladoc
annotated JobExecutionContext with @DeveloperAPI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Posted by olegz <gi...@git.apache.org>.
Github user olegz closed the pull request at:
https://github.com/apache/spark/pull/2422
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/spark/pull/2422#issuecomment-56437906
@mridulm
I do agree with Experimental so I'll change
As for the rest I am working on it so stay tuned
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2422#issuecomment-55848794
I commented on the JIRA - but this would benefit from a more complete design proposal on JIRA. Traditionally the SparkContext API has not been a point of pluggabilty. Instead we have a pluggable resource management subsystem and also pluggability in other areas such as the shuffle. So it would be helpful to understand the end-to-end proposal here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2422#issuecomment-55843559
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/2422#issuecomment-56373277
Is there an example of how this is going to be leveraged ?
The default case is the simple version delegating to existing spark - would be good to see how this is used against tez - so that there is a clearer picture of how the design changes fit into overall changes proposed : and what further changes (if any) might be required in case we need to generalize the interface.
As @pwendell mentioned, currently this aspect is not exposed - and if we want to expose it via an spi, we need to have better understanding of how it will be leveraged.
Also, would Experimental make more sense than DeveloperApi (until it stabilizes) ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org