You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Geoff Thompson <ge...@bearpeak.com> on 2014/07/09 21:42:00 UTC
how to access configuration properties on a remote Hadoop cluster
Hello,
Is there a way to query the Resource Manager for configuration properties from an external client process other than using the web interface?
Our background: We run a YARN application by running a Client on an external machine that may access one of many remote Hadoop clusters. The configuration properties on the Client machine are completely independent of the configuration properties on the remote cluster.
Currently I am getting configuration properties by using HTTP to get <Resource-Manager-WebApp-Address>/conf. However, if possible, I would prefer some form of Hadoop interface. I have looked through YarnClient and YarnClientApplication and I have not discovered anything that enables asking the Resource Manager about properties.
Does anyone have any suggestions?
Thanks,
Geoff
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Geoff Thompson <ge...@bearpeak.com>.
Hi Adam,
Thanks for the suggestion.
Geoff
On Jul 9, 2014, at 2:02 PM, Adam Kawa <ka...@gmail.com> wrote:
> Instead of <Resource-Manager-WebApp-Address>/conf, If you have application id and job id, you can query the Resource Manager for the configuration of this particular application. You can use HTTP and Java API for that.
>
>
>
> 2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an external machine that may access one of many remote Hadoop clusters. The configuration properties on the Client machine are completely independent of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get <Resource-Manager-WebApp-Address>/conf. However, if possible, I would prefer some form of Hadoop interface. I have looked through YarnClient and YarnClientApplication and I have not discovered anything that enables asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Geoff Thompson <ge...@bearpeak.com>.
Hi Adam,
Thanks for the suggestion.
Geoff
On Jul 9, 2014, at 2:02 PM, Adam Kawa <ka...@gmail.com> wrote:
> Instead of <Resource-Manager-WebApp-Address>/conf, If you have application id and job id, you can query the Resource Manager for the configuration of this particular application. You can use HTTP and Java API for that.
>
>
>
> 2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an external machine that may access one of many remote Hadoop clusters. The configuration properties on the Client machine are completely independent of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get <Resource-Manager-WebApp-Address>/conf. However, if possible, I would prefer some form of Hadoop interface. I have looked through YarnClient and YarnClientApplication and I have not discovered anything that enables asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Geoff Thompson <ge...@bearpeak.com>.
Hi Adam,
Thanks for the suggestion.
Geoff
On Jul 9, 2014, at 2:02 PM, Adam Kawa <ka...@gmail.com> wrote:
> Instead of <Resource-Manager-WebApp-Address>/conf, If you have application id and job id, you can query the Resource Manager for the configuration of this particular application. You can use HTTP and Java API for that.
>
>
>
> 2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an external machine that may access one of many remote Hadoop clusters. The configuration properties on the Client machine are completely independent of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get <Resource-Manager-WebApp-Address>/conf. However, if possible, I would prefer some form of Hadoop interface. I have looked through YarnClient and YarnClientApplication and I have not discovered anything that enables asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Geoff Thompson <ge...@bearpeak.com>.
Hi Adam,
Thanks for the suggestion.
Geoff
On Jul 9, 2014, at 2:02 PM, Adam Kawa <ka...@gmail.com> wrote:
> Instead of <Resource-Manager-WebApp-Address>/conf, If you have application id and job id, you can query the Resource Manager for the configuration of this particular application. You can use HTTP and Java API for that.
>
>
>
> 2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an external machine that may access one of many remote Hadoop clusters. The configuration properties on the Client machine are completely independent of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get <Resource-Manager-WebApp-Address>/conf. However, if possible, I would prefer some form of Hadoop interface. I have looked through YarnClient and YarnClientApplication and I have not discovered anything that enables asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Adam Kawa <ka...@gmail.com>.
Instead of <Resource-Manager-WebApp-Address>/conf, If you have application
id and job id, you can query the Resource Manager for the configuration of
this particular application. You can use HTTP and Java API for that.
2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties
> from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an
> external machine that may access one of many remote Hadoop clusters. The
> configuration properties on the Client machine are completely independent
> of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get
> <Resource-Manager-WebApp-Address>/conf. However, if possible, I would
> prefer some form of Hadoop interface. I have looked through YarnClient and
> YarnClientApplication and I have not discovered anything that enables
> asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Adam Kawa <ka...@gmail.com>.
Instead of <Resource-Manager-WebApp-Address>/conf, If you have application
id and job id, you can query the Resource Manager for the configuration of
this particular application. You can use HTTP and Java API for that.
2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties
> from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an
> external machine that may access one of many remote Hadoop clusters. The
> configuration properties on the Client machine are completely independent
> of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get
> <Resource-Manager-WebApp-Address>/conf. However, if possible, I would
> prefer some form of Hadoop interface. I have looked through YarnClient and
> YarnClientApplication and I have not discovered anything that enables
> asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Adam Kawa <ka...@gmail.com>.
Instead of <Resource-Manager-WebApp-Address>/conf, If you have application
id and job id, you can query the Resource Manager for the configuration of
this particular application. You can use HTTP and Java API for that.
2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties
> from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an
> external machine that may access one of many remote Hadoop clusters. The
> configuration properties on the Client machine are completely independent
> of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get
> <Resource-Manager-WebApp-Address>/conf. However, if possible, I would
> prefer some form of Hadoop interface. I have looked through YarnClient and
> YarnClientApplication and I have not discovered anything that enables
> asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>
Re: how to access configuration properties on a remote Hadoop cluster
Posted by Adam Kawa <ka...@gmail.com>.
Instead of <Resource-Manager-WebApp-Address>/conf, If you have application
id and job id, you can query the Resource Manager for the configuration of
this particular application. You can use HTTP and Java API for that.
2014-07-09 21:42 GMT+02:00 Geoff Thompson <ge...@bearpeak.com>:
> Hello,
>
> Is there a way to query the Resource Manager for configuration properties
> from an external client process other than using the web interface?
>
> Our background: We run a YARN application by running a Client on an
> external machine that may access one of many remote Hadoop clusters. The
> configuration properties on the Client machine are completely independent
> of the configuration properties on the remote cluster.
>
> Currently I am getting configuration properties by using HTTP to get
> <Resource-Manager-WebApp-Address>/conf. However, if possible, I would
> prefer some form of Hadoop interface. I have looked through YarnClient and
> YarnClientApplication and I have not discovered anything that enables
> asking the Resource Manager about properties.
>
> Does anyone have any suggestions?
>
> Thanks,
>
> Geoff
>
>
>