You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Scott Kuehn (JIRA)" <ji...@apache.org> on 2016/03/02 03:12:18 UTC

[jira] [Created] (SQOOP-2861) Sqoop2: Scheduler Pool Support

Scott Kuehn created SQOOP-2861:
----------------------------------

             Summary: Sqoop2: Scheduler Pool Support
                 Key: SQOOP-2861
                 URL: https://issues.apache.org/jira/browse/SQOOP-2861
             Project: Sqoop
          Issue Type: New Feature
          Components: sqoop2-framework
    Affects Versions: 2.0.0
            Reporter: Scott Kuehn


Provide a mechanism to limit cluster-wide sqoop access to a particular FROM resource. The use case is to configure a yarn scheduler pool that will limit the vcores and ram available for jobs accessing a sensitive resource. A subset of sqoop2 jobs could be configured to run in this pool, whereas other sqoop2 jobs would fall back to the default pool configured for the sqoop2 server.

The throttling extractor mechanics are useful for preventing a single job from saturating the resource, but this mechanism cannot limit aggregate resource access across jobs. This ticket aims to enable the use of scheduler pools for scenarios when multiple sqoop2 jobs would access a resource.

Possible implementation strategies:
# Enable clients to pass through job-specific mapreduce configuration, such as key=value pairs in the CLI. A sqoop2 client would specify the scheduler pool by passing a {{mapreduce.job.queuename}} from the CLI 
# Expose scheduler semantics to the client. An execution engine can subsequently decide to honor the scheduler request. For example, a pool property can be interpreted and then set as the {{mapreduce.job.queuename}} value of the hadoop configuration from the mapreduce execution engine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)