You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Manoj Samel <ma...@gmail.com> on 2015/11/04 02:12:58 UTC
Slider User Question - APIs to manage application rather than files
Hello,
First of all, apologies if this forum is for slider developers and not end
users. The slider user forum hosted @ Hortonworks community does not seem
to have any significant traffic so cross-posting here
Please take a look @
http://hortonworks.com/community/forums/topic/slider-application-deployment-using-apis/
Thanks
Manoj
Re: Slider User Question - APIs to manage application rather than
files
Posted by Steve Loughran <st...@hortonworks.com>.
> On 4 Nov 2015, at 23:00, Manoj Samel <ma...@gmail.com> wrote:
>
> Thanks Steve for the detailed update.
>
> To make sure I understood you correctly -
>
> 1. The REST API ** can be used today ** by setting
> slider.feature.ws.insecure=true.
yes. And once you do that, your code must talk directly to the URL of the Appmaster, not via the proxy. Once the flag is set, anything that can access the HTTP endpoint can manipulate the application, hence "insecure"
> This will enable update verbs like PUT and DELETE, not just GET
yes
> 2. Your recommendation is to use the slider jar as library instead of #1
>
yes, because that works in a secure cluster too. The API you can use is designed to support both a two-way IPC connection as well as the REST API (when enabled).
> Is my understanding correct?
>
yes it is.
> Thanks,
>
> On Wed, Nov 4, 2015 at 10:56 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> On 4 Nov 2015, at 01:53, Manoj Samel <manojsameltech@gmail.com<mailto:
>> manojsameltech@gmail.com>> wrote:
>>
>> Thanks Gour,
>>
>> Is there a plan to provide REST APIs in near future ?
>>
>> In use cases where configurations need to be built on fly and components be
>> added / removed on demand, the file /command line based approach does not
>> scales well.
>>
>> Thanks,
>>
>>
>> That'll be SLIDER-151, "Implement full slider API in REST and switch
>> client to it"
>>
>> All the code for this exists. Except that POST/PUT support in the slider
>> AM is disabled for security reasons.
>>
>> To ensure that only the kerberos-authenticated user can talk to slider,
>> all HTTP requests must go through the YARN RM Proxy, which does the check
>> and proxies access, also doing some HTML cleanup to prevent malicious
>> scripts & things.
>>
>> We can't go through the proxy; see YARN-2031, YARN-2084. Someone needs to
>> fix those bits of YARN.
>>
>> I've got a patch for a bit of it, but see it has aged. If someone could
>> bring that up to sync with trunk we could try to get that in to Hadoop 2.8
>> (more specifically: if people other than I write these patches, I can
>> commit it myself).
>>
>>
>> 1. You can enable the REST API by changing the HTTP filters to not
>> redirect HTTP requests sent to the ws/ bits of the REST API, which you can
>> do with by setting "slider.feature.ws.insecure" to "true" (I've done this
>> for testing).
>>
>> 2. The API is documented at :
>> http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html
>>
>> 3. The read-side of the REST model works everywhere today; we define the
>> various JSON structures in the slider JAR.
>>
>> Before rushing to turn on a back door the current UK government would be
>> proud of, know that we do have is secure API client designed to work with
>> both Slider's existing IPC protocol and the REST API. :
>> org.apache.slider.api.SliderApplicationApi
>>
>>
>> There's an RPC client ( SliderApplicationIpcClient) and REST client
>> (SliderApplicationApiRestClient); the test TestStandaloneREST shows how
>> they are interchangeable from a client perspective, they both present the
>> same rest model and operations.
>>
>> If you can put the slider JAR on your classpath, and deal with classpath
>> version pain (JAX-RS has proven the troublespot), then you can load
>> slider.jar in your app and use it as a library, rather than a command line
>> tool. This is how the Apache Ambari integration works.
>>
>>
>>
Re: Slider User Question - APIs to manage application rather than files
Posted by Manoj Samel <ma...@gmail.com>.
Thanks Steve for the detailed update.
To make sure I understood you correctly -
1. The REST API ** can be used today ** by setting
slider.feature.ws.insecure=true.
This will enable update verbs like PUT and DELETE, not just GET
2. Your recommendation is to use the slider jar as library instead of #1
Is my understanding correct?
Thanks,
On Wed, Nov 4, 2015 at 10:56 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> On 4 Nov 2015, at 01:53, Manoj Samel <manojsameltech@gmail.com<mailto:
> manojsameltech@gmail.com>> wrote:
>
> Thanks Gour,
>
> Is there a plan to provide REST APIs in near future ?
>
> In use cases where configurations need to be built on fly and components be
> added / removed on demand, the file /command line based approach does not
> scales well.
>
> Thanks,
>
>
> That'll be SLIDER-151, "Implement full slider API in REST and switch
> client to it"
>
> All the code for this exists. Except that POST/PUT support in the slider
> AM is disabled for security reasons.
>
> To ensure that only the kerberos-authenticated user can talk to slider,
> all HTTP requests must go through the YARN RM Proxy, which does the check
> and proxies access, also doing some HTML cleanup to prevent malicious
> scripts & things.
>
> We can't go through the proxy; see YARN-2031, YARN-2084. Someone needs to
> fix those bits of YARN.
>
> I've got a patch for a bit of it, but see it has aged. If someone could
> bring that up to sync with trunk we could try to get that in to Hadoop 2.8
> (more specifically: if people other than I write these patches, I can
> commit it myself).
>
>
> 1. You can enable the REST API by changing the HTTP filters to not
> redirect HTTP requests sent to the ws/ bits of the REST API, which you can
> do with by setting "slider.feature.ws.insecure" to "true" (I've done this
> for testing).
>
> 2. The API is documented at :
> http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html
>
> 3. The read-side of the REST model works everywhere today; we define the
> various JSON structures in the slider JAR.
>
> Before rushing to turn on a back door the current UK government would be
> proud of, know that we do have is secure API client designed to work with
> both Slider's existing IPC protocol and the REST API. :
> org.apache.slider.api.SliderApplicationApi
>
>
> There's an RPC client ( SliderApplicationIpcClient) and REST client
> (SliderApplicationApiRestClient); the test TestStandaloneREST shows how
> they are interchangeable from a client perspective, they both present the
> same rest model and operations.
>
> If you can put the slider JAR on your classpath, and deal with classpath
> version pain (JAX-RS has proven the troublespot), then you can load
> slider.jar in your app and use it as a library, rather than a command line
> tool. This is how the Apache Ambari integration works.
>
>
>
Re: Slider User Question - APIs to manage application rather than
files
Posted by Steve Loughran <st...@hortonworks.com>.
On 4 Nov 2015, at 01:53, Manoj Samel <ma...@gmail.com>> wrote:
Thanks Gour,
Is there a plan to provide REST APIs in near future ?
In use cases where configurations need to be built on fly and components be
added / removed on demand, the file /command line based approach does not
scales well.
Thanks,
That'll be SLIDER-151, "Implement full slider API in REST and switch client to it"
All the code for this exists. Except that POST/PUT support in the slider AM is disabled for security reasons.
To ensure that only the kerberos-authenticated user can talk to slider, all HTTP requests must go through the YARN RM Proxy, which does the check and proxies access, also doing some HTML cleanup to prevent malicious scripts & things.
We can't go through the proxy; see YARN-2031, YARN-2084. Someone needs to fix those bits of YARN.
I've got a patch for a bit of it, but see it has aged. If someone could bring that up to sync with trunk we could try to get that in to Hadoop 2.8 (more specifically: if people other than I write these patches, I can commit it myself).
1. You can enable the REST API by changing the HTTP filters to not redirect HTTP requests sent to the ws/ bits of the REST API, which you can do with by setting "slider.feature.ws.insecure" to "true" (I've done this for testing).
2. The API is documented at : http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html
3. The read-side of the REST model works everywhere today; we define the various JSON structures in the slider JAR.
Before rushing to turn on a back door the current UK government would be proud of, know that we do have is secure API client designed to work with both Slider's existing IPC protocol and the REST API. : org.apache.slider.api.SliderApplicationApi
There's an RPC client ( SliderApplicationIpcClient) and REST client (SliderApplicationApiRestClient); the test TestStandaloneREST shows how they are interchangeable from a client perspective, they both present the same rest model and operations.
If you can put the slider JAR on your classpath, and deal with classpath version pain (JAX-RS has proven the troublespot), then you can load slider.jar in your app and use it as a library, rather than a command line tool. This is how the Apache Ambari integration works.
Re: Slider User Question - APIs to manage application rather than files
Posted by Manoj Samel <ma...@gmail.com>.
Thanks Gour,
Is there a plan to provide REST APIs in near future ?
In use cases where configurations need to be built on fly and components be
added / removed on demand, the file /command line based approach does not
scales well.
Thanks,
Manoj
On Tue, Nov 3, 2015 at 5:46 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Slider does not provide REST APIs to create/manage applications yet. Until
> then a Java client application can bundle the slider-core jar and use the
> Java APIs provided in SliderClien.java.
>
>
> -Gour
>
> On 11/3/15, 5:12 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>
> >Hello,
> >
> >First of all, apologies if this forum is for slider developers and not end
> >users. The slider user forum hosted @ Hortonworks community does not seem
> >to have any significant traffic so cross-posting here
> >
> >Please take a look @
> >
> http://hortonworks.com/community/forums/topic/slider-application-deploymen
> >t-using-apis/
> >
> >Thanks
> >
> >Manoj
>
>
Re: Slider User Question - APIs to manage application rather than
files
Posted by Gour Saha <gs...@hortonworks.com>.
Slider does not provide REST APIs to create/manage applications yet. Until
then a Java client application can bundle the slider-core jar and use the
Java APIs provided in SliderClien.java.
-Gour
On 11/3/15, 5:12 PM, "Manoj Samel" <ma...@gmail.com> wrote:
>Hello,
>
>First of all, apologies if this forum is for slider developers and not end
>users. The slider user forum hosted @ Hortonworks community does not seem
>to have any significant traffic so cross-posting here
>
>Please take a look @
>http://hortonworks.com/community/forums/topic/slider-application-deploymen
>t-using-apis/
>
>Thanks
>
>Manoj