You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Mohammad Islam <mi...@yahoo.com> on 2016/11/30 01:02:23 UTC
Default Topology URL doesn't work with WebHDFS
Hi,
I set "default.app.topology.name" in gateway-site.xml to "uber" (my default topology name). It worked fine if I gave the full URL. The command looks like this "curl http://<KNOX_GW_URL>/gateway/uber/webhdfs/v1/?op=GETHOMEDIRECTORY'". However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY'". I got the HTTP error code 500. I looked into gateway.log file and found quite a few error related to rewrite. The exact error messages are shown below:
Error message
2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE
After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the following pattern and it worked for short URL but long URL faces the same issue.
Original:
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern="*://*:*/**/webhdfs/{version}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/> </rule>
Modified :
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern="*://*:*/webhdfs/{version}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/> </rule>
Overall, the rewrite pattern may be the issue. We will need to support for both short and long URL. May be, we can add multiple rewrite rules for each route in service.xml.Is there any other cleaner way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
Regards,Mohammad
Re: Default Topology URL doesn't work with WebHDFS
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Larry,It is not urgent but a convenience for our users. Now they have to append two words in the path (e.g. gateway/sandbox).
The main use case : our external services/users will connect to Kerberized Hadoop services though Knox service using corporate credential.
Regards,Mohammad
On Thursday, December 8, 2016 1:17 PM, larry mccay <lm...@apache.org> wrote:
We should certainly be able to tell whether it can be done by end of Q1, *17*. :)
If you can articulate what is driving your interest in that feature - I would appreciate it.It was always intended to help with being able to use Knox from within the cluster or with existing cli tooling that can talk directly to WebHDFS.The uptake of it has always been hindered due to some kerberos related assumptions in the java clients with regard to the WebHDFS protocol and kerberos/delegation token dance.
So, understanding another driving motivation for it would be really interesting.
On Thu, Dec 8, 2016 at 3:57 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Thanks Larry for getting back on this.This will not be a blocker. We will use the long path for the time being.
Do you think, it will be done by Q1,16?Regards,Mohammad
On Thursday, December 8, 2016 12:41 PM, larry mccay <lm...@apache.org> wrote:
Hi Mohammad -
This issue is really hard to track down.I assume it has actually been broken for quite a while.
I am curious just how important this is for your deployment and usecases.
I'd like to get to closing down other issues for the 0.11.0 release rather than have this issue delay the release - if it is something that we can get by with as a known issue for another release.
Thoughts?
thanks,
--larry
On Fri, Dec 2, 2016 at 1:08 AM, Mohammad Islam <mi...@yahoo.com.invalid> wrote:
Thanks Larry!
On Wednesday, November 30, 2016 3:45 PM, larry mccay <lm...@apache.org> wrote:
I see what's going on.This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,
I set "default.app.topology.name" in gateway-site.xml to "uber" (my default topology name). It worked fine if I gave the full URL. The command looks like this "curl http://<KNOX_GW_URL>/gateway/ uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the HTTP error code 500. I looked into gateway.log file and found quite a few error related to rewrite. The exact error messages are shown below:
Error message
2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the following pattern and it worked for short URL but long URL faces the same issue.
Original:
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Modified :
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Overall, the rewrite pattern may be the issue. We will need to support for both short and long URL. May be, we can add multiple rewrite rules for each route in service.xml.Is there any other cleaner way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
Regards,Mohammad
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
We should certainly be able to tell whether it can be done by end of Q1,
*17*. :)
If you can articulate what is driving your interest in that feature - I
would appreciate it.
It was always intended to help with being able to use Knox from within the
cluster or with existing cli tooling that can talk directly to WebHDFS.
The uptake of it has always been hindered due to some kerberos related
assumptions in the java clients with regard to the WebHDFS protocol and
kerberos/delegation token dance.
So, understanding another driving motivation for it would be really
interesting.
On Thu, Dec 8, 2016 at 3:57 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Thanks Larry for getting back on this.
> This will not be a blocker. We will use the long path for the time being.
>
> Do you think, it will be done by Q1,16?
> Regards,
> Mohammad
>
>
>
>
>
> On Thursday, December 8, 2016 12:41 PM, larry mccay <lm...@apache.org>
> wrote:
>
>
> Hi Mohammad -
>
> This issue is really hard to track down.
> I assume it has actually been broken for quite a while.
>
> I am curious just how important this is for your deployment and usecases.
>
> I'd like to get to closing down other issues for the 0.11.0 release rather
> than have this issue delay the release - if it is something that we can get
> by with as a known issue for another release.
>
> Thoughts?
>
> thanks,
>
> --larry
>
> On Fri, Dec 2, 2016 at 1:08 AM, Mohammad Islam <mislam77@yahoo.com.invalid
> > wrote:
>
> Thanks Larry!
>
>
> On Wednesday, November 30, 2016 3:45 PM, larry mccay <
> lmccay@apache.org> wrote:
>
>
> I see what's going on.This is going to take some time to debug and get a
> fix in place.
> Thanks for reporting this Mohammad!
> On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
>
> Thanks for the note, Mohammad.
> Let me play around with it and see what is going on there.Will probably
> get back to you tomorrow about it.
> On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
> Hi,
> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
> default topology name). It worked fine if I gave the full URL. The command
> looks like this "curl http://<KNOX_GW_URL>/gateway/
> uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command
> "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the
> HTTP error code 500. I looked into gateway.log file and found quite a few
> error related to rewrite. The exact error messages are shown below:
> Error message
> 2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v
> 1/?op=GETHOMEDIRECTORY, direction: IN via rule:
> WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30
> 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169))
> - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v
> 1/?op=GETHOMEDIRECTORY, direction: IN via rule:
> WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
>
> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
> following pattern and it worked for short URL but long URL faces the same
> issue.
> Original:
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root"
> pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite
> template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
> Modified :
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root"
> pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite
> template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
>
> Overall, the rewrite pattern may be the issue. We will need to support for
> both short and long URL. May be, we can add multiple rewrite rules for each
> route in service.xml.Is <http://service.xml.is/> there any other cleaner
> way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
> Regards,Mohammad
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Default Topology URL doesn't work with WebHDFS
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Thanks Larry for getting back on this.This will not be a blocker. We will use the long path for the time being.
Do you think, it will be done by Q1,16?Regards,Mohammad
On Thursday, December 8, 2016 12:41 PM, larry mccay <lm...@apache.org> wrote:
Hi Mohammad -
This issue is really hard to track down.I assume it has actually been broken for quite a while.
I am curious just how important this is for your deployment and usecases.
I'd like to get to closing down other issues for the 0.11.0 release rather than have this issue delay the release - if it is something that we can get by with as a known issue for another release.
Thoughts?
thanks,
--larry
On Fri, Dec 2, 2016 at 1:08 AM, Mohammad Islam <mi...@yahoo.com.invalid> wrote:
Thanks Larry!
On Wednesday, November 30, 2016 3:45 PM, larry mccay <lm...@apache.org> wrote:
I see what's going on.This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,
I set "default.app.topology.name" in gateway-site.xml to "uber" (my default topology name). It worked fine if I gave the full URL. The command looks like this "curl http://<KNOX_GW_URL>/gateway/ uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the HTTP error code 500. I looked into gateway.log file and found quite a few error related to rewrite. The exact error messages are shown below:
Error message
2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the following pattern and it worked for short URL but long URL faces the same issue.
Original:
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Modified :
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Overall, the rewrite pattern may be the issue. We will need to support for both short and long URL. May be, we can add multiple rewrite rules for each route in service.xml.Is there any other cleaner way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
Regards,Mohammad
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
Hi Mohammad -
This issue is really hard to track down.
I assume it has actually been broken for quite a while.
I am curious just how important this is for your deployment and usecases.
I'd like to get to closing down other issues for the 0.11.0 release rather
than have this issue delay the release - if it is something that we can get
by with as a known issue for another release.
Thoughts?
thanks,
--larry
On Fri, Dec 2, 2016 at 1:08 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Thanks Larry!
>
>
> On Wednesday, November 30, 2016 3:45 PM, larry mccay <
> lmccay@apache.org> wrote:
>
>
> I see what's going on.This is going to take some time to debug and get a
> fix in place.
> Thanks for reporting this Mohammad!
> On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
>
> Thanks for the note, Mohammad.
> Let me play around with it and see what is going on there.Will probably
> get back to you tomorrow about it.
> On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
> Hi,
> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
> default topology name). It worked fine if I gave the full URL. The command
> looks like this "curl http://<KNOX_GW_URL>/gateway/
> uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command
> "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the
> HTTP error code 500. I looked into gateway.log file and found quite a few
> error related to rewrite. The exact error messages are shown below:
> Error message
> 2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v
> 1/?op=GETHOMEDIRECTORY, direction: IN via rule:
> WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30
> 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169))
> - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v
> 1/?op=GETHOMEDIRECTORY, direction: IN via rule:
> WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
>
> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
> following pattern and it worked for short URL but long URL faces the same
> issue.
> Original:
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root"
> pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite
> template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
> Modified :
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root"
> pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite
> template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
>
> Overall, the rewrite pattern may be the issue. We will need to support for
> both short and long URL. May be, we can add multiple rewrite rules for each
> route in service.xml.Is there any other cleaner way which may work for
> all cases such as webhdfs, yarn, hive, UIs etc?
> Regards,Mohammad
>
>
>
>
>
>
>
>
>
Re: Default Topology URL doesn't work with WebHDFS
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Thanks Larry!
On Wednesday, November 30, 2016 3:45 PM, larry mccay <lm...@apache.org> wrote:
I see what's going on.This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,
I set "default.app.topology.name" in gateway-site.xml to "uber" (my default topology name). It worked fine if I gave the full URL. The command looks like this "curl http://<KNOX_GW_URL>/gateway/ uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the HTTP error code 500. I looked into gateway.log file and found quite a few error related to rewrite. The exact error messages are shown below:
Error message
2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the following pattern and it worked for short URL but long URL faces the same issue.
Original:
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Modified :
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Overall, the rewrite pattern may be the issue. We will need to support for both short and long URL. May be, we can add multiple rewrite rules for each route in service.xml.Is there any other cleaner way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
Regards,Mohammad
Re: Default Topology URL doesn't work with WebHDFS
Posted by Mohammad Islam <mi...@yahoo.com>.
Thanks Larry!
On Wednesday, November 30, 2016 3:45 PM, larry mccay <lm...@apache.org> wrote:
I see what's going on.This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,
I set "default.app.topology.name" in gateway-site.xml to "uber" (my default topology name). It worked fine if I gave the full URL. The command looks like this "curl http://<KNOX_GW_URL>/gateway/ uber/webhdfs/v1/?op=GETHOMEDIR ECTORY'". However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/ v1/?op=GETHOMEDIRECTORY'". I got the HTTP error code 500. I looked into gateway.log file and found quite a few error related to rewrite. The exact error messages are shown below:
Error message
2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE2016-11-30 00:39:51,565 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewr ite(169)) - Failed to rewrite URL: http://<KNOX_GW_URL>/webhdfs/v 1/?op=GETHOMEDIRECTORY, direction: IN via rule: WEBHDFS/webhdfs/inbound/nameno de/root, status: FAILURE
After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the following pattern and it worked for short URL but long URL faces the same issue.
Original:
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/**/webhdfs/{v ersion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Modified :
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/ namenode/root" pattern="*://*:*/webhdfs/{vers ion}/?{**}"> <rewrite template="{$serviceUrl[WEBHDFS ]}/{version}/?{**}"/> </rule>
Overall, the rewrite pattern may be the issue. We will need to support for both short and long URL. May be, we can add multiple rewrite rules for each route in service.xml.Is there any other cleaner way which may work for all cases such as webhdfs, yarn, hive, UIs etc?
Regards,Mohammad
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
I see what's going on.
This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
> Thanks for the note, Mohammad.
>
> Let me play around with it and see what is going on there.
> Will probably get back to you tomorrow about it.
>
> On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
>> Hi,
>>
>> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
>> default topology name).
>> It worked fine if I gave the full URL. The command looks like this "curl
>> http://<KNOX_GW_URL>/*gateway/uber/*webhdfs/v1/?op=GETHOMEDIRECTORY'".
>>
>> However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY'".
>> I got the HTTP error code 500. I looked into gateway.log file and found
>> quite a few error related to rewrite. The exact error messages are shown
>> below:
>>
>> *Error message*
>>
>> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
>> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
>> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
>> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
>> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>>
>>
>> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
>> following pattern and it worked for short URL but long URL faces the same
>> issue.
>>
>> *Original*:
>>
>> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern=
>> "*://*:**/**/*webhdfs/{version}/?{**}">
>> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
>> </rule>
>>
>> *Modified* :
>>
>> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root"
>> pattern="*://*:*/webhdfs/{version}/?{**}">
>> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
>> </rule>
>>
>>
>>
>> Overall, the rewrite pattern may be the issue. We will need to support
>> for both short and long URL. May be, we can add multiple rewrite rules for
>> each route in service.xml.
>> Is there any other cleaner way which may work for all cases such as
>> webhdfs, yarn, hive, UIs etc?
>>
>> Regards,
>> Mohammad
>>
>>
>>
>>
>>
>>
>
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
I see what's going on.
This is going to take some time to debug and get a fix in place.
Thanks for reporting this Mohammad!
On Tue, Nov 29, 2016 at 10:07 PM, larry mccay <lm...@apache.org> wrote:
> Thanks for the note, Mohammad.
>
> Let me play around with it and see what is going on there.
> Will probably get back to you tomorrow about it.
>
> On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
>> Hi,
>>
>> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
>> default topology name).
>> It worked fine if I gave the full URL. The command looks like this "curl
>> http://<KNOX_GW_URL>/*gateway/uber/*webhdfs/v1/?op=GETHOMEDIRECTORY'".
>>
>> However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY'".
>> I got the HTTP error code 500. I looked into gateway.log file and found
>> quite a few error related to rewrite. The exact error messages are shown
>> below:
>>
>> *Error message*
>>
>> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
>> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
>> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
>> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
>> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>>
>>
>> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
>> following pattern and it worked for short URL but long URL faces the same
>> issue.
>>
>> *Original*:
>>
>> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern=
>> "*://*:**/**/*webhdfs/{version}/?{**}">
>> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
>> </rule>
>>
>> *Modified* :
>>
>> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root"
>> pattern="*://*:*/webhdfs/{version}/?{**}">
>> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
>> </rule>
>>
>>
>>
>> Overall, the rewrite pattern may be the issue. We will need to support
>> for both short and long URL. May be, we can add multiple rewrite rules for
>> each route in service.xml.
>> Is there any other cleaner way which may work for all cases such as
>> webhdfs, yarn, hive, UIs etc?
>>
>> Regards,
>> Mohammad
>>
>>
>>
>>
>>
>>
>
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.
Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi,
>
> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
> default topology name).
> It worked fine if I gave the full URL. The command looks like this "curl
> http://<KNOX_GW_URL>/*gateway/uber/*webhdfs/v1/?op=GETHOMEDIRECTORY'".
>
> However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY'".
> I got the HTTP error code 500. I looked into gateway.log file and found
> quite a few error related to rewrite. The exact error messages are shown
> below:
>
> *Error message*
>
> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>
>
> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
> following pattern and it worked for short URL but long URL faces the same
> issue.
>
> *Original*:
>
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern=
> "*://*:**/**/*webhdfs/{version}/?{**}">
> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
> </rule>
>
> *Modified* :
>
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root"
> pattern="*://*:*/webhdfs/{version}/?{**}">
> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
> </rule>
>
>
>
> Overall, the rewrite pattern may be the issue. We will need to support for
> both short and long URL. May be, we can add multiple rewrite rules for each
> route in service.xml.
> Is there any other cleaner way which may work for all cases such as
> webhdfs, yarn, hive, UIs etc?
>
> Regards,
> Mohammad
>
>
>
>
>
>
Re: Default Topology URL doesn't work with WebHDFS
Posted by larry mccay <lm...@apache.org>.
Thanks for the note, Mohammad.
Let me play around with it and see what is going on there.
Will probably get back to you tomorrow about it.
On Tue, Nov 29, 2016 at 8:02 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi,
>
> I set "default.app.topology.name" in gateway-site.xml to "uber" (my
> default topology name).
> It worked fine if I gave the full URL. The command looks like this "curl
> http://<KNOX_GW_URL>/*gateway/uber/*webhdfs/v1/?op=GETHOMEDIRECTORY'".
>
> However, when I tried with command "curl http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY'".
> I got the HTTP error code 500. I looked into gateway.log file and found
> quite a few error related to rewrite. The exact error messages are shown
> below:
>
> *Error message*
>
> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
> *2016-11-30 00:39:51,565 ERROR hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
> http://<KNOX_GW_URL>/webhdfs/v1/?op=GETHOMEDIRECTORY, direction: IN via
> rule: WEBHDFS/webhdfs/inbound/namenode/root, status: FAILURE*
>
>
> After that, I modified the webhdfs/2.4.0/rewrite.xml by rewriting the
> following pattern and it worked for short URL but long URL faces the same
> issue.
>
> *Original*:
>
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root" pattern=
> "*://*:**/**/*webhdfs/{version}/?{**}">
> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
> </rule>
>
> *Modified* :
>
> <rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/root"
> pattern="*://*:*/webhdfs/{version}/?{**}">
> <rewrite template="{$serviceUrl[WEBHDFS]}/{version}/?{**}"/>
> </rule>
>
>
>
> Overall, the rewrite pattern may be the issue. We will need to support for
> both short and long URL. May be, we can add multiple rewrite rules for each
> route in service.xml.
> Is there any other cleaner way which may work for all cases such as
> webhdfs, yarn, hive, UIs etc?
>
> Regards,
> Mohammad
>
>
>
>
>
>