You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2023/06/15 11:42:45 UTC
[GitHub] [hadoop] slfan1989 opened a new pull request, #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
slfan1989 opened a new pull request, #5751:
URL: https://github.com/apache/hadoop/pull/5751
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
JIRA: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#issuecomment-1593036200
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 63m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/1/artifact/out/blanks-eol.txt) | The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 91m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5751 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux ec20c9fb8ab0 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d9d35c6df812ed7a94eb6faec3f0f55492ac6f84 |
| Max. process+thread count | 530 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri merged PR #5751:
URL: https://github.com/apache/hadoop/pull/5751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#discussion_r1231724814
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
Review Comment:
I will add the setup HDFS link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#discussion_r1231724192
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
+|:----------|:--------------|
+| Machine A | RM1\NM1\ZK1 | |
+| Machine B | RM2\NM2 |
+| Machine C | Router\Client |
+
+### YARN-1(ClusterTest-Yarn1)
+
+#### RM-1
+
+> For the ResourceManager, we need to configure the following option:
+
+```
+
+<!-- YARN cluster-id -->
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn1</value>
+</property>
+
+<!--
+ We can choose to use FairScheduler or CapacityScheduler. Different schedulers have different configuration.
+ FairScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
+ CapacityScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
+-->
+<property>
+ <name>yarn.resourcemanager.scheduler.class</name>
+ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
+</property>
+
+<!--
+ This configuration option is used to specify the configuration file for FairScheduler.
+ If we are using CapacityScheduler, we don't need to configure this option.
+-->
+<property>
+ <name>yarn.scheduler.fair.allocation.file</name>
+ <value>/path/fair-scheduler.xml</value>
+</property>
+
+<!-- Enable YARN Federation mode -->
+<property>
+ <name>yarn.federation.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- We use ZooKeeper to query/store Federation information. -->
+<property>
+ <name>yarn.federation.state-store.class</name>
+ <value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
+</property>
+
+<!-- ZK Address. -->
+<property>
+ <name>hadoop.zk.address</name>
+ <value>zkHost:zkPort</value>
+</property>
+
+```
+
+> Start RM
+
+```
+$HADOOP_HOME/bin/yarn --daemon start resourcemanager
+```
+
+#### NM-1
+
+> For the NodeManager, we need to configure the following option:
+
+```
+<!-- YARN cluster-id -->
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn1</value>
+</property>
+
+<!-- local dir -->
+<property>
+ <name>yarn.nodemanager.local-dirs</name>
+ <value>path/local</value>
+</property>
+
+<!-- log dir -->
+<property>
+ <name>yarn.nodemanager.log-dirs</name>
+ <value>path/logdir</value>
+</property>
+
+<!-- Enable YARN Federation mode -->
+<property>
+ <name>yarn.federation.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- Disenable YARN Federation FailOver -->
+<property>
+ <name>yarn.federation.failover.enabled</name>
+ <value>false</value>
+</property>
+
+<!-- Enable YARN Federation Non-HA Mode -->
+<property>
+ <name>yarn.federation.non-ha.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- We use ZooKeeper to query/store Federation information. -->
+<property>
+ <name>yarn.federation.state-store.class</name>
+ <value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
+</property>
+
+<!-- ZK Address. -->
+<property>
+ <name>hadoop.zk.address</name>
+ <value>zkHost:zkPort</value>
+</property>
+
+<!-- Enable AmRmProxy. -->
+<property>
+ <name>yarn.nodemanager.amrmproxy.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- interceptors to be run at the amrmproxy -->
+<property>
+ <name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>
+ <value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor</value>
+</property>
+```
+
+> Start NM
+
+```
+$HADOOP_HOME/bin/yarn --daemon start nodemanager
+```
+
+### YARN-2(ClusterTest-Yarn2)
+
+#### RM-2
+
+The RM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
+
+```
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn2</value>
+</property>
+```
+
+#### NM-2
+
+The NM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
+
+```
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn2</value>
+</property>
+```
+
+After we have finished configuring the `YARN-2` cluster, we can proceed with starting the `YARN-2` cluster.
+
+### ROUTER
Review Comment:
I will modify it to `Router`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri commented on code in PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#discussion_r1231224450
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
+|:----------|:--------------|
+| Machine A | RM1\NM1\ZK1 | |
Review Comment:
Too many |?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
+|:----------|:--------------|
+| Machine A | RM1\NM1\ZK1 | |
+| Machine B | RM2\NM2 |
+| Machine C | Router\Client |
+
+### YARN-1(ClusterTest-Yarn1)
+
+#### RM-1
+
+> For the ResourceManager, we need to configure the following option:
+
+```
+
+<!-- YARN cluster-id -->
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn1</value>
+</property>
+
+<!--
+ We can choose to use FairScheduler or CapacityScheduler. Different schedulers have different configuration.
+ FairScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
+ CapacityScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
+-->
+<property>
+ <name>yarn.resourcemanager.scheduler.class</name>
+ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
+</property>
+
+<!--
+ This configuration option is used to specify the configuration file for FairScheduler.
+ If we are using CapacityScheduler, we don't need to configure this option.
+-->
+<property>
+ <name>yarn.scheduler.fair.allocation.file</name>
+ <value>/path/fair-scheduler.xml</value>
+</property>
+
+<!-- Enable YARN Federation mode -->
+<property>
+ <name>yarn.federation.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- We use ZooKeeper to query/store Federation information. -->
+<property>
+ <name>yarn.federation.state-store.class</name>
+ <value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
+</property>
+
+<!-- ZK Address. -->
+<property>
+ <name>hadoop.zk.address</name>
+ <value>zkHost:zkPort</value>
+</property>
+
+```
+
+> Start RM
+
+```
+$HADOOP_HOME/bin/yarn --daemon start resourcemanager
+```
+
+#### NM-1
+
+> For the NodeManager, we need to configure the following option:
+
+```
+<!-- YARN cluster-id -->
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn1</value>
+</property>
+
+<!-- local dir -->
+<property>
+ <name>yarn.nodemanager.local-dirs</name>
+ <value>path/local</value>
+</property>
+
+<!-- log dir -->
+<property>
+ <name>yarn.nodemanager.log-dirs</name>
+ <value>path/logdir</value>
+</property>
+
+<!-- Enable YARN Federation mode -->
+<property>
+ <name>yarn.federation.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- Disenable YARN Federation FailOver -->
+<property>
+ <name>yarn.federation.failover.enabled</name>
+ <value>false</value>
+</property>
+
+<!-- Enable YARN Federation Non-HA Mode -->
+<property>
+ <name>yarn.federation.non-ha.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- We use ZooKeeper to query/store Federation information. -->
+<property>
+ <name>yarn.federation.state-store.class</name>
+ <value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
+</property>
+
+<!-- ZK Address. -->
+<property>
+ <name>hadoop.zk.address</name>
+ <value>zkHost:zkPort</value>
+</property>
+
+<!-- Enable AmRmProxy. -->
+<property>
+ <name>yarn.nodemanager.amrmproxy.enabled</name>
+ <value>true</value>
+</property>
+
+<!-- interceptors to be run at the amrmproxy -->
+<property>
+ <name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>
+ <value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor</value>
+</property>
+```
+
+> Start NM
+
+```
+$HADOOP_HOME/bin/yarn --daemon start nodemanager
+```
+
+### YARN-2(ClusterTest-Yarn2)
+
+#### RM-2
+
+The RM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
+
+```
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn2</value>
+</property>
+```
+
+#### NM-2
+
+The NM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
+
+```
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn2</value>
+</property>
+```
+
+After we have finished configuring the `YARN-2` cluster, we can proceed with starting the `YARN-2` cluster.
+
+### ROUTER
Review Comment:
Should we use capitals? Probably better "Router"
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
Review Comment:
Warning blank at the end of the line.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
Review Comment:
The parenthesis needs proper capitalization and the period at the end is weird.
Add link to the ZooKeeper setup.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
Review Comment:
Link to the HDFS setup markdown?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
+|:----------|:--------------|
+| Machine A | RM1\NM1\ZK1 | |
+| Machine B | RM2\NM2 |
+| Machine C | Router\Client |
+
+### YARN-1(ClusterTest-Yarn1)
+
+#### RM-1
+
+> For the ResourceManager, we need to configure the following option:
+
+```
+
+<!-- YARN cluster-id -->
+<property>
+ <name>yarn.resourcemanager.cluster-id</name>
+ <value>ClusterTest-Yarn1</value>
+</property>
+
+<!--
+ We can choose to use FairScheduler or CapacityScheduler. Different schedulers have different configuration.
+ FairScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
+ CapacityScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
+-->
+<property>
+ <name>yarn.resourcemanager.scheduler.class</name>
+ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
+</property>
+
+<!--
+ This configuration option is used to specify the configuration file for FairScheduler.
Review Comment:
Fix all the end of line blank warnings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#discussion_r1231723679
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md:
##########
@@ -385,3 +385,269 @@ The output from this particular example job should be something like:
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
+
+How to build a Test Federation Cluster
+--------------------
+
+The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
+
+Test Environment Description:
+- We need to build a HDFS test environment, this part can refer to HDFS documentation.
+- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
+- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation.
+- We need one Router and one Client.
+
+Example of Machine-Role Mapping(Exclude HDFS):
+
+| Machine | Role |
+|:----------|:--------------|
+| Machine A | RM1\NM1\ZK1 | |
Review Comment:
Thank you very much for your help in reviewing the code! I will improve this paragraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#issuecomment-1598297643
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 63m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 95m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5751 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux 485c721076ce 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 817e0cb370e835529fea501a544c50da4bb62456 |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#issuecomment-1599675346
@goiri Thank you very much for your help in reviewing the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5751: YARN-11505. [Federation] Add Steps To Set up a Test Cluster.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5751:
URL: https://github.com/apache/hadoop/pull/5751#issuecomment-1594323165
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 62m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 108m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5751 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets markdownlint |
| uname | Linux 49c1c3949e43 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dd9fee300c5745d9677ef16b6138dfe97f759f25 |
| Max. process+thread count | 623 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5751/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org