You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Sunil Parmar <su...@gmail.com> on 2017/09/26 18:53:11 UTC
how to find application master URL ( non proxy )
We're using plain vanilla version of apex in our environment ( no
datatorrent or dtgateway ). We're using STRAM API to access the application
details. We are able to find the STRAM endpoints from YARN application API.
http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING
The tracking URL property points to STRAM service but it's exposed by YARN
web proxy. i.e.
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
http://
<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/logicalPlan
http://
<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/physicalPlan
http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers
It works for monitoring but the limitation here though is that it only
allows users to submit "GET" type requests on STRAM api. So it doesn't
allow to run POST API i.e. to change log level.
How to find direct application master tracking URL ( no proxy ) ? Is there
a YARN api / STRAM api to do so ? So far I have found a brute force way to
scrap AM log to find it but I wanted to check with community experts if
there is a better way.
Thanks,
Sunil Parmar
Re: how to find application master URL ( non proxy )
Posted by Thomas Weise <th...@apache.org>.
The CLI is just another REST client (you see the calls it makes in the
source code and repeat those same calls directly). That won't solve the
problem of not being able to access the AM port directly though. Since the
RM proxy does not support POST, you would need a separate proxy mechanism
that does. Perhaps setting up an SSH tunnel with dynamic forwarding (SOCKS
proxy) will help?
On Tue, Sep 26, 2017 at 4:18 PM, Sunil Parmar <su...@gmail.com> wrote:
> Thanks ! Is CLI the only way to get it ? Is there an API to find this ? We
> have a homegrown GUI built on STRAM API and setting log levels is one of
> the desirable feature. The limitation on prod environment is that we don't
> access to all the workers nodes where potentially the AppMaster could be
> running and GUI runs from edge node.
>
> Sunil Parmar
>
> On Tue, Sep 26, 2017 at 1:23 PM, Thomas Weise <th...@apache.org> wrote:
>
>> Once the app is running you get the AM REST host/port from the app info,
>> such as:
>>
>> apex (application_1506456479682_0001) > get-app-info
>>
>> ...
>>
>> "appMasterTrackingUrl": "apex-sandbox:35463",
>>
>> ...
>>
>> Have a look at the Apex CLI or here
>> <https://github.com/atrato/atrato-server> for details on how the REST
>> calls are done.
>>
>> Thomas
>>
>> On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <su...@gmail.com>
>> wrote:
>>
>>> We're using plain vanilla version of apex in our environment ( no
>>> datatorrent or dtgateway ). We're using STRAM API to access the application
>>> details. We are able to find the STRAM endpoints from YARN application API.
>>>
>>> http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING
>>>
>>> The tracking URL property points to STRAM service but it's exposed by
>>> YARN web proxy. i.e.
>>>
>>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
>>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/
>>> stram/logicalPlan
>>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/
>>> stram/physicalPlan
>>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/
>>> stram/loggers
>>>
>>> It works for monitoring but the limitation here though is that it only
>>> allows users to submit "GET" type requests on STRAM api. So it doesn't
>>> allow to run POST API i.e. to change log level.
>>>
>>> How to find direct application master tracking URL ( no proxy ) ? Is
>>> there a YARN api / STRAM api to do so ? So far I have found a brute force
>>> way to scrap AM log to find it but I wanted to check with community experts
>>> if there is a better way.
>>>
>>> Thanks,
>>> Sunil Parmar
>>>
>>
>>
>
Re: how to find application master URL ( non proxy )
Posted by Sunil Parmar <su...@gmail.com>.
Thanks ! Is CLI the only way to get it ? Is there an API to find this ? We
have a homegrown GUI built on STRAM API and setting log levels is one of
the desirable feature. The limitation on prod environment is that we don't
access to all the workers nodes where potentially the AppMaster could be
running and GUI runs from edge node.
Sunil Parmar
On Tue, Sep 26, 2017 at 1:23 PM, Thomas Weise <th...@apache.org> wrote:
> Once the app is running you get the AM REST host/port from the app info,
> such as:
>
> apex (application_1506456479682_0001) > get-app-info
>
> ...
>
> "appMasterTrackingUrl": "apex-sandbox:35463",
>
> ...
>
> Have a look at the Apex CLI or here
> <https://github.com/atrato/atrato-server> for details on how the REST
> calls are done.
>
> Thomas
>
> On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <su...@gmail.com>
> wrote:
>
>> We're using plain vanilla version of apex in our environment ( no
>> datatorrent or dtgateway ). We're using STRAM API to access the application
>> details. We are able to find the STRAM endpoints from YARN application API.
>>
>> http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING
>>
>> The tracking URL property points to STRAM service but it's exposed by
>> YARN web proxy. i.e.
>>
>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/
>> stram/logicalPlan
>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/
>> stram/physicalPlan
>> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers
>>
>> It works for monitoring but the limitation here though is that it only
>> allows users to submit "GET" type requests on STRAM api. So it doesn't
>> allow to run POST API i.e. to change log level.
>>
>> How to find direct application master tracking URL ( no proxy ) ? Is
>> there a YARN api / STRAM api to do so ? So far I have found a brute force
>> way to scrap AM log to find it but I wanted to check with community experts
>> if there is a better way.
>>
>> Thanks,
>> Sunil Parmar
>>
>
>
Re: how to find application master URL ( non proxy )
Posted by Thomas Weise <th...@apache.org>.
Once the app is running you get the AM REST host/port from the app info,
such as:
apex (application_1506456479682_0001) > get-app-info
...
"appMasterTrackingUrl": "apex-sandbox:35463",
...
Have a look at the Apex CLI or here
<https://github.com/atrato/atrato-server> for details on how the REST calls
are done.
Thomas
On Tue, Sep 26, 2017 at 11:53 AM, Sunil Parmar <su...@gmail.com>
wrote:
> We're using plain vanilla version of apex in our environment ( no
> datatorrent or dtgateway ). We're using STRAM API to access the application
> details. We are able to find the STRAM endpoints from YARN application API.
>
> http://<RM>:8088/ws/v1/cluster/apps?state=RUNNING
>
> The tracking URL property points to STRAM service but it's exposed by YARN
> web proxy. i.e.
>
> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/
> http://<RM>:8088/proxy/application_1505865155823_
> 0039/ws/v2/stram/logicalPlan
> http://<RM>:8088/proxy/application_1505865155823_
> 0039/ws/v2/stram/physicalPlan
> http://<RM>:8088/proxy/application_1505865155823_0039/ws/v2/stram/loggers
>
> It works for monitoring but the limitation here though is that it only
> allows users to submit "GET" type requests on STRAM api. So it doesn't
> allow to run POST API i.e. to change log level.
>
> How to find direct application master tracking URL ( no proxy ) ? Is
> there a YARN api / STRAM api to do so ? So far I have found a brute force
> way to scrap AM log to find it but I wanted to check with community experts
> if there is a better way.
>
> Thanks,
> Sunil Parmar
>