[jira] [Created] (YARN-11976) [Umbrella] Add ResourceManager MCP endpoint with API key authentication

["Bence Kosztolnik (Jira)" <ji...@apache.org>]

Bence Kosztolnik created YARN-11976:
---------------------------------------

             Summary: [Umbrella] Add ResourceManager MCP endpoint with API key authentication
                 Key: YARN-11976
                 URL: https://issues.apache.org/jira/browse/YARN-11976
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: yarn
    Affects Versions: 3.6.0
            Reporter: Bence Kosztolnik
            Assignee: Bence Kosztolnik


h3. Overview

Add an optional [Model Context Protocol (MCP)|https://modelcontextprotocol.io/] endpoint to the YARN ResourceManager so agents and automation tools can query cluster state through structured MCP tools instead of ad-hoc scripts or Kerberos-bound REST calls.

The work is delivered as four stacked pull requests, each building on the previous one:
 # hadoop-common MCP framework
 # MCP API key storage and admin tooling
 # ResourceManager MCP server integration
 # Documentation

MCP is opt-in ({{{}yarn.resourcemanager.mcp.enable=false{}}} by default). On secure clusters, tool calls authenticate with admin-managed API keys stored in {{{}RMStateStore{}}}. Key administration stays on the existing Kerberos-protected RM webapp REST API and {{yarn mcpapikey}} CLI. The MCP tool endpoint runs on a dedicated port (default {{{}:8092{}}}), separate from the RM webapp ({{{}:8090{}}}).
h3. Architecture
MCP client --(API key)--> dedicated MCP server (:8092, no SPNEGO)
POST /ws/v1/mcp (initialize, tools/list, tools/call)
 
YARN admin --(Kerberos)--> RM webapp (:8090)
/ws/v1/cluster/mcp-api-keys (create/list/revoke)
Initial tools:
 * {{get_scheduler_info}}
 * {{list_applications}}

h3. Why a Hadoop-native MCP framework

The official MCP Java SDK was not used because of integration friction with Hadoop’s Jetty/Jersey/Jackson/{{{}javax.servlet{}}} stack. A minimal protocol-compatible layer lives in {{org.apache.hadoop.mcp}} in {{{}hadoop-common{}}}.
h3. Non-goals
 * Submitting or modifying YARN applications via MCP
 * OAuth/JWT or per-request Kerberos for MCP clients
 * API key expiry / automatic rotation in the server
 * Running MCP on the main RM webapp port with SPNEGO bypass

----
h2. Configuration
||Property||Default||Description||
|{{yarn.resourcemanager.mcp.enable}}|{{false}}|Master switch|
|{{yarn.resourcemanager.mcp.address}}|{{0.0.0.0:8092}}|Dedicated MCP bind address|
|{{yarn.resourcemanager.mcp.use.https}}|{{true}}|HTTPS vs plain HTTP for MCP server|
----
h2. Security notes
 * On secure clusters, only {{tools/call}} requires an API key; keys are sent in HTTP headers.
 * Plain HTTP ({{{}mcp.use.https=false{}}}) sends credentials in cleartext — startup WARN on secure clusters.
 * Default bind {{0.0.0.0}} exposes the port on all interfaces unless firewalled.
 * On unsecure clusters, {{tools/call}} is anonymous and runs as the RM process user.
 * API keys do not expire; rotation is manual (create → migrate clients → revoke).

 

Source: [https://github.com/K0K0V0K/hadoop/tree/US-YARN-MCP]

Feature doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcp.md]

Designe doc: [https://github.com/K0K0V0K/hadoop/blob/US-YARN-MCP/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerMcpDesign.md]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org