You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Scott Kuehn (JIRA)" <ji...@apache.org> on 2016/03/02 03:12:18 UTC
[jira] [Created] (SQOOP-2861) Sqoop2: Scheduler Pool Support
Scott Kuehn created SQOOP-2861:
----------------------------------
Summary: Sqoop2: Scheduler Pool Support
Key: SQOOP-2861
URL: https://issues.apache.org/jira/browse/SQOOP-2861
Project: Sqoop
Issue Type: New Feature
Components: sqoop2-framework
Affects Versions: 2.0.0
Reporter: Scott Kuehn
Provide a mechanism to limit cluster-wide sqoop access to a particular FROM resource. The use case is to configure a yarn scheduler pool that will limit the vcores and ram available for jobs accessing a sensitive resource. A subset of sqoop2 jobs could be configured to run in this pool, whereas other sqoop2 jobs would fall back to the default pool configured for the sqoop2 server.
The throttling extractor mechanics are useful for preventing a single job from saturating the resource, but this mechanism cannot limit aggregate resource access across jobs. This ticket aims to enable the use of scheduler pools for scenarios when multiple sqoop2 jobs would access a resource.
Possible implementation strategies:
# Enable clients to pass through job-specific mapreduce configuration, such as key=value pairs in the CLI. A sqoop2 client would specify the scheduler pool by passing a {{mapreduce.job.queuename}} from the CLI
# Expose scheduler semantics to the client. An execution engine can subsequently decide to honor the scheduler request. For example, a pool property can be interpreted and then set as the {{mapreduce.job.queuename}} value of the hadoop configuration from the mapreduce execution engine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)