You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Syed Shameerur Rahman (Jira)" <ji...@apache.org> on 2024/03/16 09:06:00 UTC
[jira] [Created] (YARN-11664) Remove HDFS Binaries/Jars Dependency From Yarn
Syed Shameerur Rahman created YARN-11664:
--------------------------------------------
Summary: Remove HDFS Binaries/Jars Dependency From Yarn
Key: YARN-11664
URL: https://issues.apache.org/jira/browse/YARN-11664
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn
Reporter: Syed Shameerur Rahman
In principle Hadoop Yarn is independent of HDFS. It can work with any filesystem. Currently there exists some code dependency for Yarn with HDFS. This dependency requires Yarn to bring in some of the HDFS binaries/jars to its class path. The idea behind this jira is to remove this dependency so that Yarn can run without HDFS binaries/jars
*Scope*
1. Non test classes are considered
2. Some test classes which comes as transitive dependency are considered
*Out of scope*
1. All test classes in Yarn module is not considered
--------------------------------------------------------------------------------------------
A quick search in Yarn module revealed following HDFS dependencies
1. Constants
{code:java}
import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
2. Exception
{code:java}
import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException; (Comes as a transitive dependency from DSQuotaExceededException){code}
3. Utility
{code:java}
import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
Both Yarn and HDFS depends on hadoop-common module, One straight forward approach is to move all these dependencies to hadoop-common module and both HDFS and Yarn can pick these imports.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org