You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Ashvin (JIRA)" <ji...@apache.org> on 2016/03/23 22:03:25 UTC
[jira] [Comment Edited] (REEF-1274) Add validation to
YarnConfigurationConstructor
[ https://issues.apache.org/jira/browse/REEF-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209160#comment-15209160 ]
Ashvin edited comment on REEF-1274 at 3/23/16 9:03 PM:
-------------------------------------------------------
I think to a user {{MiniYarnCluster}} is similar to full YARN setup. It is an in-process YARN cluster. Even the default config names and values are the same.
Example code to start the test cluster: {{cluster = new MiniYARNCluster(...).start()}}.
Once the test cluster starts, {{YarnConfiguration}} in the client can be constructed like this: {{new YarnConfiguration().set( "yarn.resourcemanager.address", "hostname:" + cluster.getPort() )}}. Subsequently yarn client connects to the test yarn cluster.
If {{YarnConfigurationConstructor}} could be provided with the {{String}} RM address configuration, then it could prepare {{YarnConfiguration}} just like example above. Alternatively {{YarnDriverConfiguration}} could allow binding of a custom {{YarnConfigurationConstructor}}. Given this, I think a new runtime for test cluster is an overhead.
was (Author: ashvin):
I think to a user {MiniYarnCluster} is similar to full YARN setup. It is an in-process YARN cluster. Even the default config names and values are the same.
Example code to start the test cluster: {cluster = new MiniYARNCluster(...).start()}.
Once the test cluster starts, {YarnConfiguration} in the client can be constructed like this: {new YarnConfiguration().set( "yarn.resourcemanager.address", "hostname:" + cluster.getPort() )}. Subsequently yarn client connects to the test yarn cluster.
If {YarnConfigurationConstructor} could be provided with the RM address, then it could prepare {YarnConfiguration}. Alternatively {YarnDriverConfiguration} could allow binding of a custom {YarnConfigurationConstructor}. Given this, I think a new runtime for test cluster is an overhead.
> Add validation to YarnConfigurationConstructor
> ----------------------------------------------
>
> Key: REEF-1274
> URL: https://issues.apache.org/jira/browse/REEF-1274
> Project: REEF
> Issue Type: New Feature
> Components: REEF-Runtime-YARN
> Reporter: Markus Weimer
>
> A frequent problem REEF applications encounter is a broken or unavailable YARN Configuration of the cluster. REEF can help with the diagnose of such issues as we always load / instantiate the YARN Configuration via {{o..r.runtime.yarn.util.YarnConfigurationConstructor}}. In that class, we can validate the YARN configuration and report WARNINGs into the log if something is odd. For example:
> * Is the RM IP set and reachable?
> * Is the classpath set and non-empty? If not, are the environment variables set?
> * Can a `FileSystem` instance be created using that configuration?
> There are probably more tests we can perform as we hear from our users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)