You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Bence Kosztolnik (Jira)" <ji...@apache.org> on 2026/08/05 14:31:00 UTC
[jira] [Updated] (YARN-11976) [Umbrella] Add ResourceManager MCP endpoint with API key authentication
[ https://issues.apache.org/jira/browse/YARN-11976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bence Kosztolnik updated YARN-11976:
------------------------------------
Description:
h3. Overview
Add an optional [Model Context Protocol (MCP)|https://modelcontextprotocol.io/] endpoint to the YARN ResourceManager so agents and automation tools can query cluster state through structured MCP tools instead of ad-hoc scripts or Kerberos-bound REST calls.
The work is delivered as four stacked pull requests, each building on the previous one:
# hadoop-common MCP framework
# MCP API key storage and admin tooling
# ResourceManager MCP server integration
# Documentation
MCP is opt-in ({{{}yarn.resourcemanager.mcp.enable=false{}}} by default). On secure clusters, tool calls authenticate with admin-managed API keys stored in {{{}RMStateStore{}}}. Key administration stays on the existing Kerberos-protected RM webapp REST API and {{yarn mcpapikey}} CLI. The MCP tool endpoint runs on a dedicated port (default {{{}:8092{}}}), separate from the RM webapp ({{{}:8090{}}}).
h3. Architecture
MCP client (API key) -> dedicated MCP server (:8092, no SPNEGO)
POST /ws/v1/mcp (initialize, tools/list, tools/call)
YARN admin -{-}(Kerberos){-}-> RM webapp (:8090)
/ws/v1/cluster/mcp-api-keys (create/list/revoke)
Initial tools:
* {{get_scheduler_info}}
* {{list_applications}}
h3. Why a Hadoop-native MCP framework
The official MCP Java SDK was not used because of integration friction with Hadoop’s Jetty/Jersey/Jackson/{{{}javax.servlet{}}} stack. A minimal protocol-compatible layer lives in {{org.apache.hadoop.mcp}} in {{{}hadoop-common{}}}.
h3. Non-goals
* Submitting or modifying YARN applications via MCP
* OAuth/JWT or per-request Kerberos for MCP clients
* API key expiry / automatic rotation in the server
* Running MCP on the main RM webapp port with SPNEGO bypass
----
h2. Configuration
||Property||Default||Description||
|{{yarn.resourcemanager.mcp.enable}}|{{false}}|Master switch|
|{{yarn.resourcemanager.mcp.address}}|{{0.0.0.0:8092}}|Dedicated MCP bind address|
|{{yarn.resourcemanager.mcp.use.https}}|{{true}}|HTTPS vs plain HTTP for MCP server|
----
h2. Security notes
* On secure clusters, only {{tools/call}} requires an API key; keys are sent in HTTP headers.
* Plain HTTP ({{{}mcp.use.https=false{}}}) sends credentials in cleartext — startup WARN on secure clusters.
* Default bind {{0.0.0.0}} exposes the port on all interfaces unless firewalled.
* On unsecure clusters, {{tools/call}} is anonymous and runs as the RM process user.
* API keys do not expire; rotation is manual (create → migrate clients → revoke).
Source: [https://github.com/K0K0V0K/hadoop/tree/US-YARN-MCP]
Feature doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcp.md]
Designe doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcpDesign.md]
was:
h3. Overview
Add an optional [Model Context Protocol (MCP)|https://modelcontextprotocol.io/] endpoint to the YARN ResourceManager so agents and automation tools can query cluster state through structured MCP tools instead of ad-hoc scripts or Kerberos-bound REST calls.
The work is delivered as four stacked pull requests, each building on the previous one:
# hadoop-common MCP framework
# MCP API key storage and admin tooling
# ResourceManager MCP server integration
# Documentation
MCP is opt-in ({{{}yarn.resourcemanager.mcp.enable=false{}}} by default). On secure clusters, tool calls authenticate with admin-managed API keys stored in {{{}RMStateStore{}}}. Key administration stays on the existing Kerberos-protected RM webapp REST API and {{yarn mcpapikey}} CLI. The MCP tool endpoint runs on a dedicated port (default {{{}:8092{}}}), separate from the RM webapp ({{{}:8090{}}}).
h3. Architecture
MCP client --(API key)--> dedicated MCP server (:8092, no SPNEGO)
POST /ws/v1/mcp (initialize, tools/list, tools/call)
YARN admin --(Kerberos)--> RM webapp (:8090)
/ws/v1/cluster/mcp-api-keys (create/list/revoke)
Initial tools:
* {{get_scheduler_info}}
* {{list_applications}}
h3. Why a Hadoop-native MCP framework
The official MCP Java SDK was not used because of integration friction with Hadoop’s Jetty/Jersey/Jackson/{{{}javax.servlet{}}} stack. A minimal protocol-compatible layer lives in {{org.apache.hadoop.mcp}} in {{{}hadoop-common{}}}.
h3. Non-goals
* Submitting or modifying YARN applications via MCP
* OAuth/JWT or per-request Kerberos for MCP clients
* API key expiry / automatic rotation in the server
* Running MCP on the main RM webapp port with SPNEGO bypass
----
h2. Configuration
||Property||Default||Description||
|{{yarn.resourcemanager.mcp.enable}}|{{false}}|Master switch|
|{{yarn.resourcemanager.mcp.address}}|{{0.0.0.0:8092}}|Dedicated MCP bind address|
|{{yarn.resourcemanager.mcp.use.https}}|{{true}}|HTTPS vs plain HTTP for MCP server|
----
h2. Security notes
* On secure clusters, only {{tools/call}} requires an API key; keys are sent in HTTP headers.
* Plain HTTP ({{{}mcp.use.https=false{}}}) sends credentials in cleartext — startup WARN on secure clusters.
* Default bind {{0.0.0.0}} exposes the port on all interfaces unless firewalled.
* On unsecure clusters, {{tools/call}} is anonymous and runs as the RM process user.
* API keys do not expire; rotation is manual (create → migrate clients → revoke).
Source: [https://github.com/K0K0V0K/hadoop/tree/US-YARN-MCP]
Feature doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcp.md]
Designe doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcpDesign.md]
> [Umbrella] Add ResourceManager MCP endpoint with API key authentication
> -----------------------------------------------------------------------
>
> Key: YARN-11976
> URL: https://issues.apache.org/jira/browse/YARN-11976
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.6.0
> Reporter: Bence Kosztolnik
> Assignee: Bence Kosztolnik
> Priority: Major
> Labels: MCP, yarn
>
> h3. Overview
> Add an optional [Model Context Protocol (MCP)|https://modelcontextprotocol.io/] endpoint to the YARN ResourceManager so agents and automation tools can query cluster state through structured MCP tools instead of ad-hoc scripts or Kerberos-bound REST calls.
> The work is delivered as four stacked pull requests, each building on the previous one:
> # hadoop-common MCP framework
> # MCP API key storage and admin tooling
> # ResourceManager MCP server integration
> # Documentation
> MCP is opt-in ({{{}yarn.resourcemanager.mcp.enable=false{}}} by default). On secure clusters, tool calls authenticate with admin-managed API keys stored in {{{}RMStateStore{}}}. Key administration stays on the existing Kerberos-protected RM webapp REST API and {{yarn mcpapikey}} CLI. The MCP tool endpoint runs on a dedicated port (default {{{}:8092{}}}), separate from the RM webapp ({{{}:8090{}}}).
> h3. Architecture
> MCP client (API key) -> dedicated MCP server (:8092, no SPNEGO)
> POST /ws/v1/mcp (initialize, tools/list, tools/call)
>
> YARN admin -{-}(Kerberos){-}-> RM webapp (:8090)
> /ws/v1/cluster/mcp-api-keys (create/list/revoke)
> Initial tools:
> * {{get_scheduler_info}}
> * {{list_applications}}
> h3. Why a Hadoop-native MCP framework
> The official MCP Java SDK was not used because of integration friction with Hadoop’s Jetty/Jersey/Jackson/{{{}javax.servlet{}}} stack. A minimal protocol-compatible layer lives in {{org.apache.hadoop.mcp}} in {{{}hadoop-common{}}}.
> h3. Non-goals
> * Submitting or modifying YARN applications via MCP
> * OAuth/JWT or per-request Kerberos for MCP clients
> * API key expiry / automatic rotation in the server
> * Running MCP on the main RM webapp port with SPNEGO bypass
> ----
> h2. Configuration
> ||Property||Default||Description||
> |{{yarn.resourcemanager.mcp.enable}}|{{false}}|Master switch|
> |{{yarn.resourcemanager.mcp.address}}|{{0.0.0.0:8092}}|Dedicated MCP bind address|
> |{{yarn.resourcemanager.mcp.use.https}}|{{true}}|HTTPS vs plain HTTP for MCP server|
> ----
> h2. Security notes
> * On secure clusters, only {{tools/call}} requires an API key; keys are sent in HTTP headers.
> * Plain HTTP ({{{}mcp.use.https=false{}}}) sends credentials in cleartext — startup WARN on secure clusters.
> * Default bind {{0.0.0.0}} exposes the port on all interfaces unless firewalled.
> * On unsecure clusters, {{tools/call}} is anonymous and runs as the RM process user.
> * API keys do not expire; rotation is manual (create → migrate clients → revoke).
>
> Source: [https://github.com/K0K0V0K/hadoop/tree/US-YARN-MCP]
> Feature doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcp.md]
> Designe doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcpDesign.md]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org