You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by Mohammad Islam <mi...@yahoo.com.INVALID> on 2017/04/24 07:45:05 UTC
Issue with Version 0.12 upgrade
Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access WEBHDFS through curl command, it failed with 404. See details at the end.
However, the same command worked in 0.9.1? Btw I reused the same configurations including topologies for both versions.Any idea?
Regards,Mohammad
Curl command:
$ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD" "https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept: */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2, Unknown (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
gateway.log :*******NOTE: the first line never printed for 0.9.1 case. *********
2017-04-24 07:11:33,794 DEBUG hadoop.gateway (DefaultTopologyHandler.java:handle(83)) - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795 WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to match path /gateway/poc/webhdfs/v1/tmp/janus/
Re: Issue with Version 0.12 upgrade
Posted by larry mccay <lm...@apache.org>.
I believe that code is broken, Mohammad.
There were assumptions that use to hold true that didn't make the
transition from a redirect filter to a handler which happened quite a while
back.
I also think that it is broken that we need admin and knoxsso topologies.
That needs to be triaged and fixed.
In the meantime, you can add an authorization provider config that denies
access to everyone if that is a problem.
Feel free to file JIRAs.
On Mon, Apr 24, 2017 at 3:37 PM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Thanks Sandeep.I got the info from the that link :)
> I think I found the root cause!!
> I was using only one topology file in conf/topologies directory. As soon
> as, I added admin and knoxsso, it started working.
> The associated code:On File: DefaultTopologyHandler.javaCollection<Topology>
> candidates = topologies.getTopologies(); if( candidates != null &&
> candidates.size() == 1 ) { Topology topology =
> candidates.iterator().next(); redirectContext = "/" +
> config.getGatewayPath() + "/" + topology.getName(); }
> However, one question remains , how the same thing was working for single
> topology file in 0.9.1. The above file were not changed since 0.9.0 (since
> Feb, 16).
> Another question, is the above code correct too? Looks like , it assumes
> if there is one topology just pre-append the same thing causing duplication
> in the path. Do you think we need to change this code?
> Regards,Mohammad
>
>
>
>
>
>
>
> On Monday, April 24, 2017 11:36 AM, Sandeep More <
> moresandeep@gmail.com> wrote:
>
>
> Hello Mohammad,
>
> Yup, that's the property (default.app.topology.name). The docs for this
> one
> are not very obvious :(
> Here is the doc link for this feature
> http://knox.apache.org/books/knox-0-12-0/user-guide.html#
> Default+Topology+URLs
>
> It looks like the feature broke somewhere between 0.9.1 and 0.12.
>
>
> Best,
> Sandeep
>
> On Mon, Apr 24, 2017 at 1:55 PM, Mohammad Islam <mislam77@yahoo.com.invalid
> >
> wrote:
>
> > Thanks Larry.
> > Are you talking about the property "default.app.topology.name"? I didn't
> > see any such property defined. I also grep-ed for "doc" in
> gateway-site.xml
> > without any references.
> > As I mentioned, the same conf directory worked fine for 0.9.1. but not
> > for 0.12.
> > Any other thing to debug ?
> > Regards,Mohammad
> >
> >
> > On Monday, April 24, 2017 3:34 AM, larry mccay <larry.mccay@gmail.com
> >
> > wrote:
> >
> >
> > Hi Mohammad -
> >
> > Good to hear from you!
> > Hope all is well.
> >
> > It seems that you have poc.xml configured to be the "default topology".
> > This feature isn't working properly and it seems that maybe something was
> > done to advance it towards working properly but still not quite there
> yet.
> >
> > Try removing that configuration from gateway-site.xml and try it again.
> >
> > thanks,
> >
> > --larry
> >
> >
> > On Mon, Apr 24, 2017 at 3:45 AM, Mohammad Islam
> <mislam77@yahoo.com.invalid
> > >
> > wrote:
> >
> > > Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access
> > > WEBHDFS through curl command, it failed with 404. See details at the
> end.
> > > However, the same command worked in 0.9.1? Btw I reused the same
> > > configurations including topologies for both versions.Any idea?
> > > Regards,Mohammad
> > > Curl command:
> > > $ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD"
> "
> > > https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
> > > SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS
> > > HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept:
> > > */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2,
> Unknown
> > > (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
> > >
> > > gateway.log :*******NOTE: the first line never printed for 0.9.1 case.
> > > *********
> > > 2017-04-24 07:11:33,794 DEBUG hadoop.gateway
> > (DefaultTopologyHandler.java:handle(83))
> > > - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to
> > > /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795
> > > DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received
> > > request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795
> > > WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to
> > match
> > > path /gateway/poc/webhdfs/v1/tmp/janus/
> >
> >
> >
>
>
>
>
Re: Issue with Version 0.12 upgrade
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Thanks Sandeep.I got the info from the that link :)
I think I found the root cause!!
I was using only one topology file in conf/topologies directory. As soon as, I added admin and knoxsso, it started working.
The associated code:On File: DefaultTopologyHandler.javaCollection<Topology> candidates = topologies.getTopologies(); if( candidates != null && candidates.size() == 1 ) { Topology topology = candidates.iterator().next(); redirectContext = "/" + config.getGatewayPath() + "/" + topology.getName(); }
However, one question remains , how the same thing was working for single topology file in 0.9.1. The above file were not changed since 0.9.0 (since Feb, 16).
Another question, is the above code correct too? Looks like , it assumes if there is one topology just pre-append the same thing causing duplication in the path. Do you think we need to change this code?
Regards,Mohammad
On Monday, April 24, 2017 11:36 AM, Sandeep More <mo...@gmail.com> wrote:
Hello Mohammad,
Yup, that's the property (default.app.topology.name). The docs for this one
are not very obvious :(
Here is the doc link for this feature
http://knox.apache.org/books/knox-0-12-0/user-guide.html#Default+Topology+URLs
It looks like the feature broke somewhere between 0.9.1 and 0.12.
Best,
Sandeep
On Mon, Apr 24, 2017 at 1:55 PM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Thanks Larry.
> Are you talking about the property "default.app.topology.name"? I didn't
> see any such property defined. I also grep-ed for "doc" in gateway-site.xml
> without any references.
> As I mentioned, the same conf directory worked fine for 0.9.1. but not
> for 0.12.
> Any other thing to debug ?
> Regards,Mohammad
>
>
> On Monday, April 24, 2017 3:34 AM, larry mccay <la...@gmail.com>
> wrote:
>
>
> Hi Mohammad -
>
> Good to hear from you!
> Hope all is well.
>
> It seems that you have poc.xml configured to be the "default topology".
> This feature isn't working properly and it seems that maybe something was
> done to advance it towards working properly but still not quite there yet.
>
> Try removing that configuration from gateway-site.xml and try it again.
>
> thanks,
>
> --larry
>
>
> On Mon, Apr 24, 2017 at 3:45 AM, Mohammad Islam <mislam77@yahoo.com.invalid
> >
> wrote:
>
> > Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access
> > WEBHDFS through curl command, it failed with 404. See details at the end.
> > However, the same command worked in 0.9.1? Btw I reused the same
> > configurations including topologies for both versions.Any idea?
> > Regards,Mohammad
> > Curl command:
> > $ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD" "
> > https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
> > SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS
> > HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept:
> > */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2, Unknown
> > (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
> >
> > gateway.log :*******NOTE: the first line never printed for 0.9.1 case.
> > *********
> > 2017-04-24 07:11:33,794 DEBUG hadoop.gateway
> (DefaultTopologyHandler.java:handle(83))
> > - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to
> > /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795
> > DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received
> > request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795
> > WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to
> match
> > path /gateway/poc/webhdfs/v1/tmp/janus/
>
>
>
Re: Issue with Version 0.12 upgrade
Posted by Sandeep More <mo...@gmail.com>.
Hello Mohammad,
Yup, that's the property (default.app.topology.name). The docs for this one
are not very obvious :(
Here is the doc link for this feature
http://knox.apache.org/books/knox-0-12-0/user-guide.html#Default+Topology+URLs
It looks like the feature broke somewhere between 0.9.1 and 0.12.
Best,
Sandeep
On Mon, Apr 24, 2017 at 1:55 PM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Thanks Larry.
> Are you talking about the property "default.app.topology.name"? I didn't
> see any such property defined. I also grep-ed for "doc" in gateway-site.xml
> without any references.
> As I mentioned, the same conf directory worked fine for 0.9.1. but not
> for 0.12.
> Any other thing to debug ?
> Regards,Mohammad
>
>
> On Monday, April 24, 2017 3:34 AM, larry mccay <la...@gmail.com>
> wrote:
>
>
> Hi Mohammad -
>
> Good to hear from you!
> Hope all is well.
>
> It seems that you have poc.xml configured to be the "default topology".
> This feature isn't working properly and it seems that maybe something was
> done to advance it towards working properly but still not quite there yet.
>
> Try removing that configuration from gateway-site.xml and try it again.
>
> thanks,
>
> --larry
>
>
> On Mon, Apr 24, 2017 at 3:45 AM, Mohammad Islam <mislam77@yahoo.com.invalid
> >
> wrote:
>
> > Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access
> > WEBHDFS through curl command, it failed with 404. See details at the end.
> > However, the same command worked in 0.9.1? Btw I reused the same
> > configurations including topologies for both versions.Any idea?
> > Regards,Mohammad
> > Curl command:
> > $ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD" "
> > https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
> > SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS
> > HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept:
> > */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2, Unknown
> > (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
> >
> > gateway.log :*******NOTE: the first line never printed for 0.9.1 case.
> > *********
> > 2017-04-24 07:11:33,794 DEBUG hadoop.gateway
> (DefaultTopologyHandler.java:handle(83))
> > - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to
> > /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795
> > DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received
> > request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795
> > WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to
> match
> > path /gateway/poc/webhdfs/v1/tmp/janus/
>
>
>
Re: Issue with Version 0.12 upgrade
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Thanks Larry.
Are you talking about the property "default.app.topology.name"? I didn't see any such property defined. I also grep-ed for "doc" in gateway-site.xml without any references.
As I mentioned, the same conf directory worked fine for 0.9.1. but not for 0.12.
Any other thing to debug ?
Regards,Mohammad
On Monday, April 24, 2017 3:34 AM, larry mccay <la...@gmail.com> wrote:
Hi Mohammad -
Good to hear from you!
Hope all is well.
It seems that you have poc.xml configured to be the "default topology".
This feature isn't working properly and it seems that maybe something was
done to advance it towards working properly but still not quite there yet.
Try removing that configuration from gateway-site.xml and try it again.
thanks,
--larry
On Mon, Apr 24, 2017 at 3:45 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access
> WEBHDFS through curl command, it failed with 404. See details at the end.
> However, the same command worked in 0.9.1? Btw I reused the same
> configurations including topologies for both versions.Any idea?
> Regards,Mohammad
> Curl command:
> $ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD" "
> https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
> SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS
> HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept:
> */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2, Unknown
> (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
>
> gateway.log :*******NOTE: the first line never printed for 0.9.1 case.
> *********
> 2017-04-24 07:11:33,794 DEBUG hadoop.gateway (DefaultTopologyHandler.java:handle(83))
> - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to
> /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795
> DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received
> request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795
> WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to match
> path /gateway/poc/webhdfs/v1/tmp/janus/
Re: Issue with Version 0.12 upgrade
Posted by larry mccay <la...@gmail.com>.
Hi Mohammad -
Good to hear from you!
Hope all is well.
It seems that you have poc.xml configured to be the "default topology".
This feature isn't working properly and it seems that maybe something was
done to advance it towards working properly but still not quite there yet.
Try removing that configuration from gateway-site.xml and try it again.
thanks,
--larry
On Mon, Apr 24, 2017 at 3:45 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Hi,I recently upgraded Knox from 0.9.1 to 0.12. When I tried to access
> WEBHDFS through curl command, it failed with 404. See details at the end.
> However, the same command worked in 0.9.1? Btw I reused the same
> configurations including topologies for both versions.Any idea?
> Regards,Mohammad
> Curl command:
> $ curl -ivk -H "X-Params-Email: Xyz@abcf.com" -H "X-Own-Source: ABCD" "
> https://HOSTNAME:8443/gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS"
> SSLv2, Unknown (23):> GET /gateway/poc/webhdfs/v1/tmp/?op=LISTSTATUS
> HTTP/1.1> User-Agent: curl/7.38.0> Host: HOSTNAME:8443> Accept:
> */*> X-Params-Email: Xyz@abcd.com> X-Own-Source: ABCD> * SSLv2, Unknown
> (23):< HTTP/1.1 404 Not FoundHTTP/1.1 404 Not Found
>
> gateway.log :*******NOTE: the first line never printed for 0.9.1 case.
> *********
> 2017-04-24 07:11:33,794 DEBUG hadoop.gateway (DefaultTopologyHandler.java:handle(83))
> - Default topology forward from /gateway/poc/webhdfs/v1/tmp/janus/ to
> /gateway/poc/gateway/poc/webhdfs/v1/tmp/janus/ 2017-04-24 07:11:33,795
> DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received
> request: GET /gateway/poc/webhdfs/v1/tmp/janus/2017-04-24 07:11:33,795
> WARN hadoop.gateway (GatewayFilter.java:doFilter(162)) - Failed to match
> path /gateway/poc/webhdfs/v1/tmp/janus/