You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Theyaa Matti <th...@gmail.com> on 2018/09/04 18:09:11 UTC
Knox wrong query param encoding help
I am having issues with Knox encoding the following URL when parsing html
content.
/proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
This URL is generated by Yarn to track a running spark job. When applying
regular rules to the above URL, Knox encodes the URL to:
/proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
Which makes the URL unusable and leads to a 404. To make the URL work I
have to remove "amp%3B" and then it works.
Below are the filter and the rule I am using for this URL and I appreciate
your help/comments for a remediation.
<filter name="YARNUI/yarn/outbound/apps1">
<content type="*/html">
<apply path="/proxy/[^']+/stages/stage[^']*"
rule="YARNUI/yarn/outbound/apps/history3"/>
</content>
</filter>
<rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
<match pattern="/proxy/{*}/stages/stage?{**}"/>
<rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
</rule>
Thank you for your help.
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
Sure thank you David. The issue seemed to be that the request object gets
corrupted and becomes null by the time it reaches my custom dispatch class.
You can find my code above for your reference.
On Tue, Sep 25, 2018 at 5:24 PM David Morin <mo...@gmail.com>
wrote:
> ok :(
> So, I'll work on it on my side this week
> I keep you in touch if I succeed in fixing that issue
>
> Le mar. 25 sept. 2018 à 23:10, Theyaa Matti <th...@gmail.com> a
> écrit :
>
>> No, I am waiting for the Knox dev team to provide a solution or
>> suggestion to try.
>>
>> I followed all provided suggestions and nothing worked
>>
>> Sent from my iPhone
>>
>> On Sep 25, 2018, at 4:41 PM, David Morin <mo...@gmail.com>
>> wrote:
>>
>> I've faced to the same issue with the execution of Spark jobs on Yarn.
>> 404 error with url that contains this: "amp%3B"
>> Have you succeeded in resolving that ?
>>
>> Regards,
>> David
>>
>> Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <th...@gmail.com> a
>> écrit :
>>
>>> I have made the necessary modifications to use my custom dispatcher.
>>> However I am getting NPE from HttpServletRequest.getRequestURL() when the
>>> queryString contains special characters like "amp%3B". In this case the
>>> request object is null.
>>>
>>> Any suggestions?
>>>
>>> Regards,
>>>
>>> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> You can extend RMUIHaDispatch class and override getDispatchUrl()
>>>> method from URLDecodingDispatch
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes I am using HA for resource manager and in that case what should I
>>>>> do to the fix this issue?
>>>>>
>>>>> Best,
>>>>>
>>>>> Theyaa.
>>>>>
>>>>>
>>>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Are you using HA setup ? in which case org.apache.
>>>>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I created the class as follows:
>>>>>>>
>>>>>>> package org.apache.knox.gateway.dispatch;
>>>>>>>
>>>>>>> import javax.servlet.http.HttpServletRequest;
>>>>>>> import java.net.URI;
>>>>>>> import java.net.URLDecoder;
>>>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>>>
>>>>>>> /**
>>>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>>>> * This is useful in cases where the url is picked up
>>>>>>> * from the query parameter and is already encoded.
>>>>>>> *
>>>>>>> * @since 1.1.0
>>>>>>> */
>>>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>>>
>>>>>>> public URLDecodingDispatch() {
>>>>>>> super();
>>>>>>> }
>>>>>>>
>>>>>>> @Override
>>>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>>>> String decoded;
>>>>>>>
>>>>>>> try {
>>>>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>>>> } catch (final Exception e) {
>>>>>>> /* fall back in case of exception */
>>>>>>> decoded = request.getRequestURL().toString();
>>>>>>> }
>>>>>>>
>>>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>>>> final String query = request.getQueryString();
>>>>>>> if ( query != null ) {
>>>>>>> str.append('?');
>>>>>>> str.append(query);
>>>>>>> }
>>>>>>> final URI url = URI.create(str.toString());
>>>>>>> return url;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can
>>>>>>> see Knox loading the jar file at start:
>>>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) -
>>>>>>> ALIAS abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>>>
>>>>>>> I modified the service.xml for yarnui as follows.
>>>>>>>
>>>>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Deleted everything under deployments and restarted knox.
>>>>>>>
>>>>>>>
>>>>>>> I do not see any changes in the behavior and do not see that class
>>>>>>> being called. Do you please know what I am missing?
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> That should be a good idea, but I still have an issue with adding
>>>>>>>>> the custom dispatch to the yarnui service.xml. I already have an existing
>>>>>>>>> class there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>>>> ha-classname=
>>>>>>>>> "org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>>>
>>>>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>>>>> there?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> In that case you might have to build the jar yourself or
>>>>>>>>>> preferably use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Dhruv
>>>>>>>>>>
>>>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <
>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry 0.12.0
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <
>>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <
>>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> You can try
>>>>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job.
>>>>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the
>>>>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and
>>>>>>>>>>>>>> I appreciate your help/comments for a remediation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>>>> </content>
>>>>>>>>>>>>>> </filter>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>>> </rule>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by David Morin <mo...@gmail.com>.
ok :(
So, I'll work on it on my side this week
I keep you in touch if I succeed in fixing that issue
Le mar. 25 sept. 2018 à 23:10, Theyaa Matti <th...@gmail.com> a
écrit :
> No, I am waiting for the Knox dev team to provide a solution or suggestion
> to try.
>
> I followed all provided suggestions and nothing worked
>
> Sent from my iPhone
>
> On Sep 25, 2018, at 4:41 PM, David Morin <mo...@gmail.com>
> wrote:
>
> I've faced to the same issue with the execution of Spark jobs on Yarn.
> 404 error with url that contains this: "amp%3B"
> Have you succeeded in resolving that ?
>
> Regards,
> David
>
> Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <th...@gmail.com> a
> écrit :
>
>> I have made the necessary modifications to use my custom dispatcher.
>> However I am getting NPE from HttpServletRequest.getRequestURL() when the
>> queryString contains special characters like "amp%3B". In this case the
>> request object is null.
>>
>> Any suggestions?
>>
>> Regards,
>>
>> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> You can extend RMUIHaDispatch class and override getDispatchUrl()
>>> method from URLDecodingDispatch
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> Yes I am using HA for resource manager and in that case what should I
>>>> do to the fix this issue?
>>>>
>>>> Best,
>>>>
>>>> Theyaa.
>>>>
>>>>
>>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Are you using HA setup ? in which case org.apache.
>>>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>>
>>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I created the class as follows:
>>>>>>
>>>>>> package org.apache.knox.gateway.dispatch;
>>>>>>
>>>>>> import javax.servlet.http.HttpServletRequest;
>>>>>> import java.net.URI;
>>>>>> import java.net.URLDecoder;
>>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>>
>>>>>> /**
>>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>>> * This is useful in cases where the url is picked up
>>>>>> * from the query parameter and is already encoded.
>>>>>> *
>>>>>> * @since 1.1.0
>>>>>> */
>>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>>
>>>>>> public URLDecodingDispatch() {
>>>>>> super();
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>>> String decoded;
>>>>>>
>>>>>> try {
>>>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>>> } catch (final Exception e) {
>>>>>> /* fall back in case of exception */
>>>>>> decoded = request.getRequestURL().toString();
>>>>>> }
>>>>>>
>>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>>> final String query = request.getQueryString();
>>>>>> if ( query != null ) {
>>>>>> str.append('?');
>>>>>> str.append(query);
>>>>>> }
>>>>>> final URI url = URI.create(str.toString());
>>>>>> return url;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>>>> Knox loading the jar file at start:
>>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>>
>>>>>> I modified the service.xml for yarnui as follows.
>>>>>>
>>>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Deleted everything under deployments and restarted knox.
>>>>>>
>>>>>>
>>>>>> I do not see any changes in the behavior and do not see that class
>>>>>> being called. Do you please know what I am missing?
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> That should be a good idea, but I still have an issue with adding
>>>>>>>> the custom dispatch to the yarnui service.xml. I already have an existing
>>>>>>>> class there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"
>>>>>>>> />
>>>>>>>>
>>>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>>>> there?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> In that case you might have to build the jar yourself or
>>>>>>>>> preferably use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Dhruv
>>>>>>>>>
>>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <
>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry 0.12.0
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <
>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <
>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> You can try
>>>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job.
>>>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the
>>>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and
>>>>>>>>>>>>> I appreciate your help/comments for a remediation.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>>> </content>
>>>>>>>>>>>>> </filter>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> </rule>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
No, I am waiting for the Knox dev team to provide a solution or suggestion to try.
I followed all provided suggestions and nothing worked
Sent from my iPhone
> On Sep 25, 2018, at 4:41 PM, David Morin <mo...@gmail.com> wrote:
>
> I've faced to the same issue with the execution of Spark jobs on Yarn.
> 404 error with url that contains this: "amp%3B"
> Have you succeeded in resolving that ?
>
> Regards,
> David
>
>> Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <th...@gmail.com> a écrit :
>> I have made the necessary modifications to use my custom dispatcher. However I am getting NPE from HttpServletRequest.getRequestURL() when the queryString contains special characters like "amp%3B". In this case the request object is null.
>>
>> Any suggestions?
>>
>> Regards,
>>
>>> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com> wrote:
>>> You can extend RMUIHaDispatch class and override getDispatchUrl() method from URLDecodingDispatch
>>>
>>> Best,
>>> Sandeep
>>>
>>>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com> wrote:
>>>> Yes I am using HA for resource manager and in that case what should I do to the fix this issue?
>>>>
>>>> Best,
>>>>
>>>> Theyaa.
>>>>
>>>>
>>>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com> wrote:
>>>>> Are you using HA setup ? in which case org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>>
>>>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com> wrote:
>>>>>> I created the class as follows:
>>>>>> package org.apache.knox.gateway.dispatch;
>>>>>>
>>>>>> import javax.servlet.http.HttpServletRequest;
>>>>>> import java.net.URI;
>>>>>> import java.net.URLDecoder;
>>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>>
>>>>>> /**
>>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>>> * This is useful in cases where the url is picked up
>>>>>> * from the query parameter and is already encoded.
>>>>>> *
>>>>>> * @since 1.1.0
>>>>>> */
>>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>> public URLDecodingDispatch() {
>>>>>> super();
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>>> String decoded;
>>>>>>
>>>>>> try {
>>>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>>> } catch (final Exception e) {
>>>>>> /* fall back in case of exception */
>>>>>> decoded = request.getRequestURL().toString();
>>>>>> }
>>>>>>
>>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>>> final String query = request.getQueryString();
>>>>>> if ( query != null ) {
>>>>>> str.append('?');
>>>>>> str.append(query);
>>>>>> }
>>>>>> final URI url = URI.create(str.toString());
>>>>>> return url;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see Knox loading the jar file at start:
>>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>>
>>>>>> I modified the service.xml for yarnui as follows.
>>>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch" ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>>
>>>>>> Deleted everything under deployments and restarted knox.
>>>>>>
>>>>>> I do not see any changes in the behavior and do not see that class being called. Do you please know what I am missing?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com> wrote:
>>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com> wrote:
>>>>>>>> That should be a good idea, but I still have an issue with adding the custom dispatch to the yarnui service.xml. I already have an existing class there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch" ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>>
>>>>>>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com> wrote:
>>>>>>>>> In that case you might have to build the jar yourself or preferably use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Dhruv
>>>>>>>>>
>>>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com> wrote:
>>>>>>>>>> Sorry 0.12.0
>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com> wrote:
>>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com> wrote:
>>>>>>>>>>>> You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in service.xml ( Knox 1.1.0)
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com> wrote:
>>>>>>>>>>>>> I am having issues with Knox encoding the following URL when parsing html content.
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I appreciate your help/comments for a remediation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>>> </content>
>>>>>>>>>>>>> </filter>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> </rule>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by David Morin <mo...@gmail.com>.
I've faced to the same issue with the execution of Spark jobs on Yarn.
404 error with url that contains this: "amp%3B"
Have you succeeded in resolving that ?
Regards,
David
Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <th...@gmail.com> a écrit :
> I have made the necessary modifications to use my custom dispatcher.
> However I am getting NPE from HttpServletRequest.getRequestURL() when the
> queryString contains special characters like "amp%3B". In this case the
> request object is null.
>
> Any suggestions?
>
> Regards,
>
> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com> wrote:
>
>> You can extend RMUIHaDispatch class and override getDispatchUrl() method
>> from URLDecodingDispatch
>>
>> Best,
>> Sandeep
>>
>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> Yes I am using HA for resource manager and in that case what should I do
>>> to the fix this issue?
>>>
>>> Best,
>>>
>>> Theyaa.
>>>
>>>
>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> Are you using HA setup ? in which case org.apache.
>>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>
>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> I created the class as follows:
>>>>>
>>>>> package org.apache.knox.gateway.dispatch;
>>>>>
>>>>> import javax.servlet.http.HttpServletRequest;
>>>>> import java.net.URI;
>>>>> import java.net.URLDecoder;
>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>
>>>>> /**
>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>> * This is useful in cases where the url is picked up
>>>>> * from the query parameter and is already encoded.
>>>>> *
>>>>> * @since 1.1.0
>>>>> */
>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>
>>>>> public URLDecodingDispatch() {
>>>>> super();
>>>>> }
>>>>>
>>>>> @Override
>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>> String decoded;
>>>>>
>>>>> try {
>>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>> } catch (final Exception e) {
>>>>> /* fall back in case of exception */
>>>>> decoded = request.getRequestURL().toString();
>>>>> }
>>>>>
>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>> final String query = request.getQueryString();
>>>>> if ( query != null ) {
>>>>> str.append('?');
>>>>> str.append(query);
>>>>> }
>>>>> final URI url = URI.create(str.toString());
>>>>> return url;
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>>> Knox loading the jar file at start:
>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>
>>>>> I modified the service.xml for yarnui as follows.
>>>>>
>>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>
>>>>>
>>>>>
>>>>> Deleted everything under deployments and restarted knox.
>>>>>
>>>>>
>>>>> I do not see any changes in the behavior and do not see that class
>>>>> being called. Do you please know what I am missing?
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> That should be a good idea, but I still have an issue with adding
>>>>>>> the custom dispatch to the yarnui service.xml. I already have an existing
>>>>>>> class there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"
>>>>>>> />
>>>>>>>
>>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>>> there?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Dhruv
>>>>>>>>
>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <
>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry 0.12.0
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> You can try
>>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Sandeep
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job.
>>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the
>>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>> </content>
>>>>>>>>>>>> </filter>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>> </rule>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
I have made the necessary modifications to use my custom dispatcher.
However I am getting NPE from HttpServletRequest.getRequestURL() when the
queryString contains special characters like "amp%3B". In this case the
request object is null.
Any suggestions?
Regards,
On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com> wrote:
> You can extend RMUIHaDispatch class and override getDispatchUrl() method
> from URLDecodingDispatch
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> Yes I am using HA for resource manager and in that case what should I do
>> to the fix this issue?
>>
>> Best,
>>
>> Theyaa.
>>
>>
>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> Are you using HA setup ? in which case org.apache.
>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>
>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> I created the class as follows:
>>>>
>>>> package org.apache.knox.gateway.dispatch;
>>>>
>>>> import javax.servlet.http.HttpServletRequest;
>>>> import java.net.URI;
>>>> import java.net.URLDecoder;
>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>
>>>> /**
>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>> * This is useful in cases where the url is picked up
>>>> * from the query parameter and is already encoded.
>>>> *
>>>> * @since 1.1.0
>>>> */
>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>
>>>> public URLDecodingDispatch() {
>>>> super();
>>>> }
>>>>
>>>> @Override
>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>> String decoded;
>>>>
>>>> try {
>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>> } catch (final Exception e) {
>>>> /* fall back in case of exception */
>>>> decoded = request.getRequestURL().toString();
>>>> }
>>>>
>>>> final StringBuffer str = new StringBuffer(decoded);
>>>> final String query = request.getQueryString();
>>>> if ( query != null ) {
>>>> str.append('?');
>>>> str.append(query);
>>>> }
>>>> final URI url = URI.create(str.toString());
>>>> return url;
>>>> }
>>>> }
>>>>
>>>>
>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>> Knox loading the jar file at start:
>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>
>>>> I modified the service.xml for yarnui as follows.
>>>>
>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>
>>>>
>>>>
>>>> Deleted everything under deployments and restarted knox.
>>>>
>>>>
>>>> I do not see any changes in the behavior and do not see that class
>>>> being called. Do you please know what I am missing?
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> That should be a good idea, but I still have an issue with adding the
>>>>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>>>>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>> there?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>
>>>>>>> Regards
>>>>>>> Dhruv
>>>>>>>
>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry 0.12.0
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> You can try
>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>> parsing html content.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>> </content>
>>>>>>>>>>> </filter>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
The problem is with Yarn not generating correct html entities. Below is the
URL generated by yarn
<a href="http://host:port/gateway/yarnui/yarn/proxy/application_222222_2222/stages/stage?amp%3Battempt=0&id=0"
class="name-link">
As you can see, there is no & before amp%3 which prevents the URLDecoder
from properly decoding it and just leave it there causing the url to become
invalid. The class you have in Knox 1.1.0 does not properly decode this url
and the result of using it is:
http://host:port
/gateway/yarnui/yarn/proxy/application_222222_2222/stages/stage?amp;attempt=0&id=0
Regards,
On Thu, Sep 6, 2018 at 3:52 PM Theyaa Matti <th...@gmail.com> wrote:
> Thank you Sandeep for the quick reply. I created the new class and
> modified services.xml for yarnui to point to it.
>
> I could see the class being loaded in the logs but the url encoding is the
> same, which tells me that the url is not passing through the new class.
>
> Here is the dispatch tag from service.xml
>
> <dispatch classname="com.gm.hadoop.knox.URLDecodingDispatch" ha-classname="
> org.apache.knox.gateway.dispatch.RMHaURLDecodingDispatch"/>
>
>
> and Here is the code for that classs
>
>
> package org.apache.knox.gateway.dispatch;
>
> import org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch;
>
> import javax.servlet.http.HttpServletRequest;
> import java.net.URI;
> import java.net.URLDecoder;
> import javax.servlet.ServletException;
>
> /**
> * Dispatch which decodes the outgoing URLs (to services).
> * This is useful in cases where the url is picked up
> * from the query parameter and is already encoded.
> *
> * @since 1.1.0
> */
> public class RMHaURLDecodingDispatch extends RMUIHaDispatch {
>
> public RMHaURLDecodingDispatch() throws ServletException{
> super();
> }
>
> @Override
> public URI getDispatchUrl(final HttpServletRequest request) {
> String decoded;
>
> try {
> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
> } catch (final Exception e) {
> /* fall back in case of exception */
> decoded = request.getRequestURL().toString();
> }
>
> final StringBuffer str = new StringBuffer(decoded);
> final String query = request.getQueryString();
> if ( query != null ) {
> str.append('?');
> str.append(query);
> }
> final URI url = URI.create(str.toString());
> return url;
> }
> }
>
>
> Regards,
>
>
>
> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com> wrote:
>
>> You can extend RMUIHaDispatch class and override getDispatchUrl() method
>> from URLDecodingDispatch
>>
>> Best,
>> Sandeep
>>
>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> Yes I am using HA for resource manager and in that case what should I do
>>> to the fix this issue?
>>>
>>> Best,
>>>
>>> Theyaa.
>>>
>>>
>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> Are you using HA setup ? in which case org.apache.
>>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>>
>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> I created the class as follows:
>>>>>
>>>>> package org.apache.knox.gateway.dispatch;
>>>>>
>>>>> import javax.servlet.http.HttpServletRequest;
>>>>> import java.net.URI;
>>>>> import java.net.URLDecoder;
>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>
>>>>> /**
>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>> * This is useful in cases where the url is picked up
>>>>> * from the query parameter and is already encoded.
>>>>> *
>>>>> * @since 1.1.0
>>>>> */
>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>
>>>>> public URLDecodingDispatch() {
>>>>> super();
>>>>> }
>>>>>
>>>>> @Override
>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>> String decoded;
>>>>>
>>>>> try {
>>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>> } catch (final Exception e) {
>>>>> /* fall back in case of exception */
>>>>> decoded = request.getRequestURL().toString();
>>>>> }
>>>>>
>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>> final String query = request.getQueryString();
>>>>> if ( query != null ) {
>>>>> str.append('?');
>>>>> str.append(query);
>>>>> }
>>>>> final URI url = URI.create(str.toString());
>>>>> return url;
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>>> Knox loading the jar file at start:
>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>
>>>>> I modified the service.xml for yarnui as follows.
>>>>>
>>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>
>>>>>
>>>>>
>>>>> Deleted everything under deployments and restarted knox.
>>>>>
>>>>>
>>>>> I do not see any changes in the behavior and do not see that class
>>>>> being called. Do you please know what I am missing?
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> That should be a good idea, but I still have an issue with adding
>>>>>>> the custom dispatch to the yarnui service.xml. I already have an existing
>>>>>>> class there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"
>>>>>>> />
>>>>>>>
>>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>>> there?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Dhruv
>>>>>>>>
>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <
>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry 0.12.0
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> You can try
>>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Sandeep
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job.
>>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the
>>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>> </content>
>>>>>>>>>>>> </filter>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>> </rule>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
Thank you Sandeep for the quick reply. I created the new class and modified
services.xml for yarnui to point to it.
I could see the class being loaded in the logs but the url encoding is the
same, which tells me that the url is not passing through the new class.
Here is the dispatch tag from service.xml
<dispatch classname="com.gm.hadoop.knox.URLDecodingDispatch" ha-classname="
org.apache.knox.gateway.dispatch.RMHaURLDecodingDispatch"/>
and Here is the code for that classs
package org.apache.knox.gateway.dispatch;
import org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch;
import javax.servlet.http.HttpServletRequest;
import java.net.URI;
import java.net.URLDecoder;
import javax.servlet.ServletException;
/**
* Dispatch which decodes the outgoing URLs (to services).
* This is useful in cases where the url is picked up
* from the query parameter and is already encoded.
*
* @since 1.1.0
*/
public class RMHaURLDecodingDispatch extends RMUIHaDispatch {
public RMHaURLDecodingDispatch() throws ServletException{
super();
}
@Override
public URI getDispatchUrl(final HttpServletRequest request) {
String decoded;
try {
decoded =
URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
} catch (final Exception e) {
/* fall back in case of exception */
decoded = request.getRequestURL().toString();
}
final StringBuffer str = new StringBuffer(decoded);
final String query = request.getQueryString();
if ( query != null ) {
str.append('?');
str.append(query);
}
final URI url = URI.create(str.toString());
return url;
}
}
Regards,
On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <mo...@gmail.com> wrote:
> You can extend RMUIHaDispatch class and override getDispatchUrl() method
> from URLDecodingDispatch
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> Yes I am using HA for resource manager and in that case what should I do
>> to the fix this issue?
>>
>> Best,
>>
>> Theyaa.
>>
>>
>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> Are you using HA setup ? in which case org.apache.
>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>
>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> I created the class as follows:
>>>>
>>>> package org.apache.knox.gateway.dispatch;
>>>>
>>>> import javax.servlet.http.HttpServletRequest;
>>>> import java.net.URI;
>>>> import java.net.URLDecoder;
>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>
>>>> /**
>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>> * This is useful in cases where the url is picked up
>>>> * from the query parameter and is already encoded.
>>>> *
>>>> * @since 1.1.0
>>>> */
>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>
>>>> public URLDecodingDispatch() {
>>>> super();
>>>> }
>>>>
>>>> @Override
>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>> String decoded;
>>>>
>>>> try {
>>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>> } catch (final Exception e) {
>>>> /* fall back in case of exception */
>>>> decoded = request.getRequestURL().toString();
>>>> }
>>>>
>>>> final StringBuffer str = new StringBuffer(decoded);
>>>> final String query = request.getQueryString();
>>>> if ( query != null ) {
>>>> str.append('?');
>>>> str.append(query);
>>>> }
>>>> final URI url = URI.create(str.toString());
>>>> return url;
>>>> }
>>>> }
>>>>
>>>>
>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>> Knox loading the jar file at start:
>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>
>>>> I modified the service.xml for yarnui as follows.
>>>>
>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>
>>>>
>>>>
>>>> Deleted everything under deployments and restarted knox.
>>>>
>>>>
>>>> I do not see any changes in the behavior and do not see that class
>>>> being called. Do you please know what I am missing?
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> That should be a good idea, but I still have an issue with adding the
>>>>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>>>>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>> there?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>
>>>>>>> Regards
>>>>>>> Dhruv
>>>>>>>
>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry 0.12.0
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> You can try
>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>> theyaamatti@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>> parsing html content.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>> </content>
>>>>>>>>>>> </filter>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Sandeep Moré <mo...@gmail.com>.
You can extend RMUIHaDispatch class and override getDispatchUrl() method
from URLDecodingDispatch
Best,
Sandeep
On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <th...@gmail.com> wrote:
> Yes I am using HA for resource manager and in that case what should I do
> to the fix this issue?
>
> Best,
>
> Theyaa.
>
>
> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com> wrote:
>
>> Are you using HA setup ? in which case org.apache.
>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>
>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> I created the class as follows:
>>>
>>> package org.apache.knox.gateway.dispatch;
>>>
>>> import javax.servlet.http.HttpServletRequest;
>>> import java.net.URI;
>>> import java.net.URLDecoder;
>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>
>>> /**
>>> * Dispatch which decodes the outgoing URLs (to services).
>>> * This is useful in cases where the url is picked up
>>> * from the query parameter and is already encoded.
>>> *
>>> * @since 1.1.0
>>> */
>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>
>>> public URLDecodingDispatch() {
>>> super();
>>> }
>>>
>>> @Override
>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>> String decoded;
>>>
>>> try {
>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>> } catch (final Exception e) {
>>> /* fall back in case of exception */
>>> decoded = request.getRequestURL().toString();
>>> }
>>>
>>> final StringBuffer str = new StringBuffer(decoded);
>>> final String query = request.getQueryString();
>>> if ( query != null ) {
>>> str.append('?');
>>> str.append(query);
>>> }
>>> final URI url = URI.create(str.toString());
>>> return url;
>>> }
>>> }
>>>
>>>
>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>> Knox loading the jar file at start:
>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>
>>> I modified the service.xml for yarnui as follows.
>>>
>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>>
>>>
>>> Deleted everything under deployments and restarted knox.
>>>
>>>
>>> I do not see any changes in the behavior and do not see that class being
>>> called. Do you please know what I am missing?
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> That should be a good idea, but I still have an issue with adding the
>>>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>>>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>
>>>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>
>>>>>> Regards
>>>>>> Dhruv
>>>>>>
>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry 0.12.0
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am using Knox 1.12.0
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You can try
>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Sandeep
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>> parsing html content.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>> <content type="*/html">
>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>> </content>
>>>>>>>>>> </filter>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>> </rule>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you for your help.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
Yes I am using HA for resource manager and in that case what should I do to
the fix this issue?
Best,
Theyaa.
On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <mo...@gmail.com> wrote:
> Are you using HA setup ? in which case org.apache.
> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>
> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> I created the class as follows:
>>
>> package org.apache.knox.gateway.dispatch;
>>
>> import javax.servlet.http.HttpServletRequest;
>> import java.net.URI;
>> import java.net.URLDecoder;
>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>
>> /**
>> * Dispatch which decodes the outgoing URLs (to services).
>> * This is useful in cases where the url is picked up
>> * from the query parameter and is already encoded.
>> *
>> * @since 1.1.0
>> */
>> public class URLDecodingDispatch extends DefaultDispatch {
>>
>> public URLDecodingDispatch() {
>> super();
>> }
>>
>> @Override
>> public URI getDispatchUrl(final HttpServletRequest request) {
>> String decoded;
>>
>> try {
>> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>> } catch (final Exception e) {
>> /* fall back in case of exception */
>> decoded = request.getRequestURL().toString();
>> }
>>
>> final StringBuffer str = new StringBuffer(decoded);
>> final String query = request.getQueryString();
>> if ( query != null ) {
>> str.append('?');
>> str.append(query);
>> }
>> final URI url = URI.create(str.toString());
>> return url;
>> }
>> }
>>
>>
>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>> Knox loading the jar file at start:
>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>
>> I modified the service.xml for yarnui as follows.
>>
>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>
>>
>>
>> Deleted everything under deployments and restarted knox.
>>
>>
>> I do not see any changes in the behavior and do not see that class being
>> called. Do you please know what I am missing?
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> That should be a good idea, but I still have an issue with adding the
>>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>
>>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>>
>>>> Regards,
>>>>
>>>>
>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com>
>>>> wrote:
>>>>
>>>>> In that case you might have to build the jar yourself or preferably
>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>
>>>>> Regards
>>>>> Dhruv
>>>>>
>>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry 0.12.0
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am using Knox 1.12.0
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You can try
>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>> parsing html content.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /proxy/application_222233323323
>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>> <content type="*/html">
>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>> </content>
>>>>>>>>> </filter>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>> </rule>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thank you for your help.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Sandeep Moré <mo...@gmail.com>.
Are you using HA setup ? in which case org.apache.
hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <th...@gmail.com> wrote:
> I created the class as follows:
>
> package org.apache.knox.gateway.dispatch;
>
> import javax.servlet.http.HttpServletRequest;
> import java.net.URI;
> import java.net.URLDecoder;
> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>
> /**
> * Dispatch which decodes the outgoing URLs (to services).
> * This is useful in cases where the url is picked up
> * from the query parameter and is already encoded.
> *
> * @since 1.1.0
> */
> public class URLDecodingDispatch extends DefaultDispatch {
>
> public URLDecodingDispatch() {
> super();
> }
>
> @Override
> public URI getDispatchUrl(final HttpServletRequest request) {
> String decoded;
>
> try {
> decoded = URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
> } catch (final Exception e) {
> /* fall back in case of exception */
> decoded = request.getRequestURL().toString();
> }
>
> final StringBuffer str = new StringBuffer(decoded);
> final String query = request.getQueryString();
> if ( query != null ) {
> str.append('?');
> str.append(query);
> }
> final URI url = URI.create(str.toString());
> return url;
> }
> }
>
>
> packaged it into a jar and uploaded it to the knox lib dir. I can see Knox
> loading the jar file at start:
> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>
> I modified the service.xml for yarnui as follows.
>
> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>
>
>
> Deleted everything under deployments and restarted knox.
>
>
> I do not see any changes in the behavior and do not see that class being
> called. Do you please know what I am missing?
>
>
> Regards,
>
>
>
>
>
>
> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com> wrote:
>
>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>
>> Best,
>> Sandeep
>>
>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> That should be a good idea, but I still have an issue with adding the
>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>
>>> Regards,
>>>
>>>
>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com> wrote:
>>>
>>>> In that case you might have to build the jar yourself or preferably use
>>>> your custom dispatch similar to URLDecodingDispatch.
>>>>
>>>> Regards
>>>> Dhruv
>>>>
>>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry 0.12.0
>>>>>
>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am using Knox 1.12.0
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> You can try
>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>> parsing html content.
>>>>>>>>
>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>
>>>>>>>>
>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>
>>>>>>>>
>>>>>>>> /proxy/application_222233323323
>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>
>>>>>>>>
>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>
>>>>>>>>
>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>
>>>>>>>>
>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>> <content type="*/html">
>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>> </content>
>>>>>>>> </filter>
>>>>>>>>
>>>>>>>>
>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>> </rule>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for your help.
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
I created the class as follows:
package org.apache.knox.gateway.dispatch;
import javax.servlet.http.HttpServletRequest;
import java.net.URI;
import java.net.URLDecoder;
import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
/**
* Dispatch which decodes the outgoing URLs (to services).
* This is useful in cases where the url is picked up
* from the query parameter and is already encoded.
*
* @since 1.1.0
*/
public class URLDecodingDispatch extends DefaultDispatch {
public URLDecodingDispatch() {
super();
}
@Override
public URI getDispatchUrl(final HttpServletRequest request) {
String decoded;
try {
decoded =
URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
} catch (final Exception e) {
/* fall back in case of exception */
decoded = request.getRequestURL().toString();
}
final StringBuffer str = new StringBuffer(decoded);
final String query = request.getQueryString();
if ( query != null ) {
str.append('?');
str.append(query);
}
final URI url = URI.create(str.toString());
return url;
}
}
packaged it into a jar and uploaded it to the knox lib dir. I can see Knox
loading the jar file at start:
resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
can=/knox/lib/knox-1.0-SNAPSHOT.jar
I modified the service.xml for yarnui as follows.
<dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
Deleted everything under deployments and restarted knox.
I do not see any changes in the behavior and do not see that class being
called. Do you please know what I am missing?
Regards,
On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <mo...@gmail.com> wrote:
> URLDecodingDispatch extends DefaultDispatch so you should be there.
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com> wrote:
>
>> That should be a good idea, but I still have an issue with adding the
>> custom dispatch to the yarnui service.xml. I already have an existing class
>> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>
>> Should I replace the DefaultDispatch or I can have more than one there?
>>
>> Regards,
>>
>>
>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com> wrote:
>>
>>> In that case you might have to build the jar yourself or preferably use
>>> your custom dispatch similar to URLDecodingDispatch.
>>>
>>> Regards
>>> Dhruv
>>>
>>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> Sorry 0.12.0
>>>>
>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am using Knox 1.12.0
>>>>>
>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You can try
>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>> service.xml ( Knox 1.1.0)
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am having issues with Knox encoding the following URL when parsing
>>>>>>> html content.
>>>>>>>
>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>
>>>>>>>
>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>
>>>>>>>
>>>>>>> /proxy/application_222233323323
>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>
>>>>>>>
>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>
>>>>>>>
>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>
>>>>>>>
>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>> <content type="*/html">
>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>> </content>
>>>>>>> </filter>
>>>>>>>
>>>>>>>
>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>>
>>>>>>> Thank you for your help.
>>>>>>>
>>>>>>>
>>>>>>>
Re: Knox wrong query param encoding help
Posted by Sandeep Moré <mo...@gmail.com>.
URLDecodingDispatch extends DefaultDispatch so you should be there.
Best,
Sandeep
On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <th...@gmail.com> wrote:
> That should be a good idea, but I still have an issue with adding the
> custom dispatch to the yarnui service.xml. I already have an existing class
> there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>
> Should I replace the DefaultDispatch or I can have more than one there?
>
> Regards,
>
>
> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com> wrote:
>
>> In that case you might have to build the jar yourself or preferably use
>> your custom dispatch similar to URLDecodingDispatch.
>>
>> Regards
>> Dhruv
>>
>> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> Sorry 0.12.0
>>>
>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> I am using Knox 1.12.0
>>>>
>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> You can try
>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>> service.xml ( Knox 1.1.0)
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am having issues with Knox encoding the following URL when parsing
>>>>>> html content.
>>>>>>
>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>
>>>>>>
>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>
>>>>>>
>>>>>> /proxy/application_222233323323
>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>
>>>>>>
>>>>>> Which makes the URL unusable and leads to a 404. To make the URL work
>>>>>> I have to remove "amp%3B" and then it works.
>>>>>>
>>>>>>
>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>> appreciate your help/comments for a remediation.
>>>>>>
>>>>>>
>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>> <content type="*/html">
>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>> </content>
>>>>>> </filter>
>>>>>>
>>>>>>
>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>> </rule>
>>>>>>
>>>>>>
>>>>>> Thank you for your help.
>>>>>>
>>>>>>
>>>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
That should be a good idea, but I still have an issue with adding the
custom dispatch to the yarnui service.xml. I already have an existing class
there <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
Should I replace the DefaultDispatch or I can have more than one there?
Regards,
On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <77...@gmail.com> wrote:
> In that case you might have to build the jar yourself or preferably use
> your custom dispatch similar to URLDecodingDispatch.
>
> Regards
> Dhruv
>
> On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com>
> wrote:
>
>> Sorry 0.12.0
>>
>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> I am using Knox 1.12.0
>>>
>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> You can try
>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>> service.xml ( Knox 1.1.0)
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am having issues with Knox encoding the following URL when parsing
>>>>> html content.
>>>>>
>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>
>>>>>
>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>
>>>>>
>>>>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>>>>
>>>>>
>>>>> Which makes the URL unusable and leads to a 404. To make the URL work
>>>>> I have to remove "amp%3B" and then it works.
>>>>>
>>>>>
>>>>> Below are the filter and the rule I am using for this URL and I
>>>>> appreciate your help/comments for a remediation.
>>>>>
>>>>>
>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>> <content type="*/html">
>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>> </content>
>>>>> </filter>
>>>>>
>>>>>
>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>> </rule>
>>>>>
>>>>>
>>>>> Thank you for your help.
>>>>>
>>>>>
>>>>>
Re: Knox wrong query param encoding help
Posted by Dhruv Goyal <77...@gmail.com>.
In that case you might have to build the jar yourself or preferably use
your custom dispatch similar to URLDecodingDispatch.
Regards
Dhruv
On Wednesday, September 5, 2018, Theyaa Matti <th...@gmail.com> wrote:
> Sorry 0.12.0
>
> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> I am using Knox 1.12.0
>>
>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>> dispatch in service.xml ( Knox 1.1.0)
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>>> wrote:
>>>
>>>> I am having issues with Knox encoding the following URL when parsing
>>>> html content.
>>>>
>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>
>>>>
>>>> This URL is generated by Yarn to track a running spark job. When
>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>
>>>>
>>>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>>>
>>>>
>>>> Which makes the URL unusable and leads to a 404. To make the URL work I
>>>> have to remove "amp%3B" and then it works.
>>>>
>>>>
>>>> Below are the filter and the rule I am using for this URL and I
>>>> appreciate your help/comments for a remediation.
>>>>
>>>>
>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>> <content type="*/html">
>>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>>> </content>
>>>> </filter>
>>>>
>>>>
>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>> </rule>
>>>>
>>>>
>>>> Thank you for your help.
>>>>
>>>>
>>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
Sorry 0.12.0
On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <th...@gmail.com> wrote:
> I am using Knox 1.12.0
>
> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com> wrote:
>
>> You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>> dispatch in service.xml ( Knox 1.1.0)
>>
>> Best,
>> Sandeep
>>
>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> I am having issues with Knox encoding the following URL when parsing
>>> html content.
>>>
>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>
>>>
>>> This URL is generated by Yarn to track a running spark job. When
>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>
>>>
>>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>>
>>>
>>> Which makes the URL unusable and leads to a 404. To make the URL work I
>>> have to remove "amp%3B" and then it works.
>>>
>>>
>>> Below are the filter and the rule I am using for this URL and I
>>> appreciate your help/comments for a remediation.
>>>
>>>
>>> <filter name="YARNUI/yarn/outbound/apps1">
>>> <content type="*/html">
>>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>>> </content>
>>> </filter>
>>>
>>>
>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>> </rule>
>>>
>>>
>>> Thank you for your help.
>>>
>>>
>>>
Re: Knox wrong query param encoding help
Posted by Theyaa Matti <th...@gmail.com>.
I am using Knox 1.12.0
On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <mo...@gmail.com> wrote:
> You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
> dispatch in service.xml ( Knox 1.1.0)
>
> Best,
> Sandeep
>
> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> I am having issues with Knox encoding the following URL when parsing html
>> content.
>>
>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>
>>
>> This URL is generated by Yarn to track a running spark job. When applying
>> regular rules to the above URL, Knox encodes the URL to:
>>
>>
>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>
>>
>> Which makes the URL unusable and leads to a 404. To make the URL work I
>> have to remove "amp%3B" and then it works.
>>
>>
>> Below are the filter and the rule I am using for this URL and I
>> appreciate your help/comments for a remediation.
>>
>>
>> <filter name="YARNUI/yarn/outbound/apps1">
>> <content type="*/html">
>> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
>> </content>
>> </filter>
>>
>>
>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>> </rule>
>>
>>
>> Thank you for your help.
>>
>>
>>
Re: Knox wrong query param encoding help
Posted by Sandeep Moré <mo...@gmail.com>.
You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
dispatch in service.xml ( Knox 1.1.0)
Best,
Sandeep
On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <th...@gmail.com> wrote:
> I am having issues with Knox encoding the following URL when parsing html
> content.
>
> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>
>
> This URL is generated by Yarn to track a running spark job. When applying
> regular rules to the above URL, Knox encodes the URL to:
>
>
> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>
>
> Which makes the URL unusable and leads to a 404. To make the URL work I
> have to remove "amp%3B" and then it works.
>
>
> Below are the filter and the rule I am using for this URL and I appreciate
> your help/comments for a remediation.
>
>
> <filter name="YARNUI/yarn/outbound/apps1">
> <content type="*/html">
> <apply path="/proxy/[^']+/stages/stage[^']*" rule="YARNUI/yarn/outbound/apps/history3"/>
> </content>
> </filter>
>
>
> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
> <match pattern="/proxy/{*}/stages/stage?{**}"/>
> <rewrite template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
> </rule>
>
>
> Thank you for your help.
>
>
>