You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by Jeff Ramsdale <je...@earthlink.net> on 2003/09/11 09:45:13 UTC
wsproxy and request parameters
I'm playing with the Amazon.com web service toolkit and Cocoon.
Here's my sitemap snippete:
<map:match pattern="doSearch.html">
<map:generate name="wsproxy"
src="http://xml.amazon.com/onca/xml3?t=webservices-20&dev-t=[my dev
code]&ActorSearch={request-param:actor}&mode=dvd&type=lite&p
age=1&f=xml" />
<map:transform src="stylesheets/lite-data-to-html.xsl" />
<map:serialize type="xhtml" />
</map:match>
When I request my page with
http://localhost:8888/Amazon/doSearch.html?actor=johnny+depp I don't get any
results back from Amazon. When I replace "{request-param:actor}" in my
sitemap with "johnny%20depp" I get a bunch of results back.
Any hints on why my parameters aren't being passed along? I'm on a pretty
recent CVS checkout of Cocoon.
Thanks,
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Tony Collen <co...@umn.edu>.
Jeff Ramsdale wrote:
> I discovered that changing my search from "johnny depp" to "depp" returns a
> list of results. The problem, then, would seem to be the encoding of spaces
> in my request parameters. When I submit my search form it converts the space
> to a "+" in my browser. Examining the logs I see that they show " ", not "+"
> on my failed searches. As mentioned before, in my sitemap if I change
> "{request-param:actor}" to "johnny%20depp" I also get a list of results. So,
> how do I ensure that my web service request is being sent with "%20" instead
> of " " or "+"?
Jeff,
I've noticed this too, a while ago.... it is a little frustrating not being able to have it change
the + to a %20... perhaps this type of encoding behavior needs to be added to the wsproxy.
Anybody else have thoughts? What should the proper encoding of this be?
Tony
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
> Jeff Ramsdale wrote:
>
> <snip/>
>
> > Actually, I figured that was probably the case. So Tony's
> example must have
> > been an error. I wanted to trust him! ;-)
>
> Yes, I messed up.. all those &'s should read as &. I was lazy and
> stupid.
Well, let's not overdo it! I though the winky would lighten my jibe--I'm
sorry if it offended!
> Anyway, the short of it is that the proxy takes the "provided" request
> parameters, and combines them with the request parameters that were
> submitted via the form, and combines them. I believe the provided
> request params are not being encoded. I was partially into solving the
> problem when I realized the {raw-request-param:actor} was not correctly
> working.
Ah, ok. That makes sense. Looks like this must not be a popular feature if
no one's discovered this before. (Or maybe no one's looked deeply enough and
all the people that ran into it left in disgust...) Anyway, you seem to be
making progress and you offered a simple work-around. Kudos to you! Let me
know if I can be of help...
> I'll check it out more tomorrow and come up with something.
>
> Tony
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Tony Collen <co...@umn.edu>.
Jeff Ramsdale wrote:
<snip/>
> Actually, I figured that was probably the case. So Tony's example must have
> been an error. I wanted to trust him! ;-)
Yes, I messed up.. all those &'s should read as &. I was lazy and
stupid.
Anyway, the short of it is that the proxy takes the "provided" request
parameters, and combines them with the request parameters that were
submitted via the form, and combines them. I believe the provided
request params are not being encoded. I was partially into solving the
problem when I realized the {raw-request-param:actor} was not correctly
working.
I'll check it out more tomorrow and come up with something.
Tony
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
Joerg,
> >><map:generate
> >> type="wsproxy"
> >> src="http://foo.bar/endpoint?parm1=val1&parm2=val2"/>
> >
> > Early on I noticed trouble here when testing and manually
> changed all the &
> > characters to & in my src string. The example you show didn't do
> > that--does it work for you as written above? Just checked--I
> get the error:
> > The reference to entity "dev-t" must end with the ';' delimiter.
> >
> > So it would seem that manual encoding is necessary in the src
> attribute? I
> > even get the error while editing my sitemap in Eclipse with the SunBow
> > plug-in, so this is enforced by the schema, it seems...
>
> That's a must in XML, it has nothing to do with the sitemap, a dtd or a
> schema. & is a reserved character in XML (for entities => so your error
> message) and must be escaped by &.
Actually, I figured that was probably the case. So Tony's example must have
been an error. I wanted to trust him! ;-)
> Joerg
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Joerg Heinicke <jh...@virbus.de>.
Jeff Ramsdale wrote:
>
>>The first is from the "src" attribute, like so:
>>
>><map:generate
>> type="wsproxy"
>> src="http://foo.bar/endpoint?parm1=val1&parm2=val2"/>
>
> Early on I noticed trouble here when testing and manually changed all the &
> characters to & in my src string. The example you show didn't do
> that--does it work for you as written above? Just checked--I get the error:
> The reference to entity "dev-t" must end with the ';' delimiter.
>
> So it would seem that manual encoding is necessary in the src attribute? I
> even get the error while editing my sitemap in Eclipse with the SunBow
> plug-in, so this is enforced by the schema, it seems...
That's a must in XML, it has nothing to do with the sitemap, a dtd or a
schema. & is a reserved character in XML (for entities => so your error
message) and must be escaped by &.
Joerg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
Hi again,
> > Worked great! Thanks!
>
> Yes, but I've re-opened the bug, because it really isn't re-encoding the
> request parameters correctly. The WSProxy, along with the newer
> HttpProxy, get request parameters from two different sources.
>
> The first is from the "src" attribute, like so:
>
> <map:generate
> type="wsproxy"
> src="http://foo.bar/endpoint?parm1=val1&parm2=val2"/>
>
> The second is from any parameters that were passed to the matching
> pipeline. All of the parameters are gathered together and then shipped
> off to the remote server in one go. Check out the fetch() method of the
> WebServiceProxyGenerator.
Early on I noticed trouble here when testing and manually changed all the &
characters to & in my src string. The example you show didn't do
that--does it work for you as written above? Just checked--I get the error:
The reference to entity "dev-t" must end with the ';' delimiter.
So it would seem that manual encoding is necessary in the src attribute? I
even get the error while editing my sitemap in Eclipse with the SunBow
plug-in, so this is enforced by the schema, it seems...
> The problem is that the parameters in the first example, which are
> included in the src attribute, are not being correctly encoded. I have
> yet to fix this, but I'll work on it. The WSProxy will probably
> eventually go away in favor of the HttpProxyGenerator, soon anyway.
Can you explain the latter a little more? I don't really know the relative
merits of the two...
<snip>
> > Interesting. What happens if I don't want to pass my parameters along?
>
> The other problem that I found is when you want to "rename" the request
> parameters which are then passed to the wsproxy. (When you don't want to
> pass all the parameters along or you wish to rename them).
>
> In this case, you would have something like this:
>
> <map:match pattern="formSubmit">
> <map:generate type="file"
> src="http://foo.com/service?parm1=val1&parm2={request-param:fo
> oParam}"/>
> <map:serialize type="xml"/>
> </map:match>
>
> I noticed there was a "RawRequestParameterModule" which provides the
> "undecoded" request parameter. Your specific problem I believe was that
> the use of {request-param:actor} was causing the parameter to be decoded
> with the space (to "johnny depp"), and then you were directly sending
> that string with the space to the remote service, which is incorrect.
> The correct use would be {raw-request-param:actor}.
That makes sense, though I'm not sure I've seen it work correctly yet.
When I submit my form it encodes my parameters with a +:
http://localhost:8888/Amazon/doRestSearch.html?actor=johnny+depp
and I get 0 results.
I observed early on that my request only worked when the URL was written out
with a %20 in place of the +. So I'm wondering if raw-request-param converts
it appropriately.
> Note that you don't actually need the WSProxy in your case, since it's
> all XML over HTTP (the FileGenerator can handle that just fine), and you
> don't neccesarily want to pass all of your form variables directly
> through to Amazon.
That makes sense now that I see my parameters are passed transparently. I
have yet to see type="file" work correctly. Changing wsproxy to file I've
got:
<map:generate type="file"
src="http://xml.amazon.com/onca/xml3?t=webservices-20&dev-t=[dev-token]&
amp;{raw-request-param:actor}&mode=dvd&type=lite&page=1&f=xm
l" />
I get 0 results. Changing the wsproxy to file in your working example also
produces 0 results. Is there something else that has to change besides the
type when using the FileGenerator?
So, in summary, passing the parameter transparently through (as in your
first example) worked great. I have not yet used request-param or
raw-request-param with any success.
> Yep, it's taken me a while to get back into digging through everything
> again. I suppose for now, use the FileGenerator, along with the
> raw-request-param InputModule. I ran into problems getting the WSProxy
> to encode stuff, even stuff coming out of the raw-request-module. I
> suspect this bug also occurs inside the HttpProxy, so I will have a look
> at both. Part of my problems are from my inexperience with
> commons-httpclient, which both the WSProxy and the HttpProxy use.
I'd like to follow your suggestion, if I could get it to work! I'll start
checking the logs for errors if you don't see an obvious syntactical problem
above.
> Tony
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Tony Collen <co...@umn.edu>.
Jeff Ramsdale wrote:
>>From: Tony Collen [mailto:colle006@umn.edu]
<snip/>
>>Jeff,
>>
>>Could you try the following test and tell me if you get search results?
>
>
> Worked great! Thanks!
Yes, but I've re-opened the bug, because it really isn't re-encoding the
request parameters correctly. The WSProxy, along with the newer
HttpProxy, get request parameters from two different sources.
The first is from the "src" attribute, like so:
<map:generate
type="wsproxy"
src="http://foo.bar/endpoint?parm1=val1&parm2=val2"/>
The second is from any parameters that were passed to the matching
pipeline. All of the parameters are gathered together and then shipped
off to the remote server in one go. Check out the fetch() method of the
WebServiceProxyGenerator.
The problem is that the parameters in the first example, which are
included in the src attribute, are not being correctly encoded. I have
yet to fix this, but I'll work on it. The WSProxy will probably
eventually go away in favor of the HttpProxyGenerator, soon anyway.
<snip/>
>>I think what is happening in your example is that when you use
>>{request-param:actor}, that is being
>>decoded by the RequestParameterInputModule, and is therefore
>>being sent unencoded to Amazon (I could
>>be wrong on this). The WSProxy automatically sends all request
>>parameters which were sent from the
>>original form, so if you have a field called "ActorSearch" in
>>your field, the WSProxy will also send
>>ActorSearch=foo to the remote service.
>
>
> Interesting. What happens if I don't want to pass my parameters along?
The other problem that I found is when you want to "rename" the request
parameters which are then passed to the wsproxy. (When you don't want to
pass all the parameters along or you wish to rename them).
In this case, you would have something like this:
<map:match pattern="formSubmit">
<map:generate type="file"
src="http://foo.com/service?parm1=val1&parm2={request-param:fooParam}"/>
<map:serialize type="xml"/>
</map:match>
I noticed there was a "RawRequestParameterModule" which provides the
"undecoded" request parameter. Your specific problem I believe was that
the use of {request-param:actor} was causing the parameter to be decoded
with the space (to "johnny depp"), and then you were directly sending
that string with the space to the remote service, which is incorrect.
The correct use would be {raw-request-param:actor}.
Note that you don't actually need the WSProxy in your case, since it's
all XML over HTTP (the FileGenerator can handle that just fine), and you
don't neccesarily want to pass all of your form variables directly
through to Amazon.
> I wish I understood the Cocoon code better. I'd like to take a look at
> it--especially with your help. I see you've posted a couple of other things
> today, so I'm not sure if you've moved on from this proposed solution. In
> any case I'm all ears...
Yep, it's taken me a while to get back into digging through everything
again. I suppose for now, use the FileGenerator, along with the
raw-request-param InputModule. I ran into problems getting the WSProxy
to encode stuff, even stuff coming out of the raw-request-module. I
suspect this bug also occurs inside the HttpProxy, so I will have a look
at both. Part of my problems are from my inexperience with
commons-httpclient, which both the WSProxy and the HttpProxy use.
> Jeff
Tony
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
> From: Tony Collen [mailto:colle006@umn.edu]
> > It hadn't occured to me that this was a problem with wsproxy. I
> just thought
> > I needed to do something to convert the format. But then I
> realized that I
> > had to manually change my all my "&"s to "&"s. Which format
> should the
> > user be expected to use for the src attribute of the generator?
> It seems to
> > me the more human-readable form makes the most sense, with encoding
> > happening in the wsproxy.
>
>
> Jeff,
>
> Could you try the following test and tell me if you get search results?
Worked great! Thanks!
> In your sitemap (Note the YOUR_AMAZON_KEY in the global-variables):
>
> <map:pipelines>
>
> <map:component-configurations>
> <global-variables>
> <amazon-key>YOUR_AMAZON_KEY</amazon-key>
> </global-variables>
> </map:component-configurations>
Elegant. I like it...
> <map:pipeline>
> <map:match pattern="*.html">
> <map:read src="{1}.html" mime-type="text/html"/>
> </map:match>
>
> <map:match pattern="actorSearch">
> <map:generate type="wsproxy"
> src="http://xml.amazon.com/onca/xml3?t=webservices-20&dev-t={g
> lobal:amazon-key}&&mode=vhs&type=lite&page=1&f=xml"/>
> <map:serialize type="xml"/>
> </map:match>
>
> </map:pipeline>
>
> </map:pipelines>
Ok, interesting. You took the parameters out. Must be passed implicitly...
> And then use the following form.html file:
>
> <html><body>
> <form method="post" action="actorSearch">
> <input type="text" name="ActorSearch" />
> <input type="submit"/>
> </form>
> </body></html>
>
> I think what is happening in your example is that when you use
> {request-param:actor}, that is being
> decoded by the RequestParameterInputModule, and is therefore
> being sent unencoded to Amazon (I could
> be wrong on this). The WSProxy automatically sends all request
> parameters which were sent from the
> original form, so if you have a field called "ActorSearch" in
> your field, the WSProxy will also send
> ActorSearch=foo to the remote service.
Interesting. What happens if I don't want to pass my parameters along?
> This is nice, but it's not always useful if you don't want to use
> Amazon's form field names, so I
> can see why you might have wanted to use the different input name
> in your form. AFAIK, it's up to
> the "sender" of the URL to make sure that request parameter names
> and values are URL-encoded, so I
> am debating whether this might warrant an input module which
> performs URL encoding.
>
> Thoughts?
I wish I understood the Cocoon code better. I'd like to take a look at
it--especially with your help. I see you've posted a couple of other things
today, so I'm not sure if you've moved on from this proposed solution. In
any case I'm all ears...
> Regards,
Thanks again for digging into this...
> Tony
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Tony Collen <co...@umn.edu>.
Jeff Ramsdale wrote:
<snip/>
> It hadn't occured to me that this was a problem with wsproxy. I just thought
> I needed to do something to convert the format. But then I realized that I
> had to manually change my all my "&"s to "&"s. Which format should the
> user be expected to use for the src attribute of the generator? It seems to
> me the more human-readable form makes the most sense, with encoding
> happening in the wsproxy.
Jeff,
Could you try the following test and tell me if you get search results?
In your sitemap (Note the YOUR_AMAZON_KEY in the global-variables):
<map:pipelines>
<map:component-configurations>
<global-variables>
<amazon-key>YOUR_AMAZON_KEY</amazon-key>
</global-variables>
</map:component-configurations>
<map:pipeline>
<map:match pattern="*.html">
<map:read src="{1}.html" mime-type="text/html"/>
</map:match>
<map:match pattern="actorSearch">
<map:generate type="wsproxy"
src="http://xml.amazon.com/onca/xml3?t=webservices-20&dev-t={global:amazon-key}&&mode=vhs&type=lite&page=1&f=xml"/>
<map:serialize type="xml"/>
</map:match>
</map:pipeline>
</map:pipelines>
And then use the following form.html file:
<html><body>
<form method="post" action="actorSearch">
<input type="text" name="ActorSearch" />
<input type="submit"/>
</form>
</body></html>
I think what is happening in your example is that when you use {request-param:actor}, that is being
decoded by the RequestParameterInputModule, and is therefore being sent unencoded to Amazon (I could
be wrong on this). The WSProxy automatically sends all request parameters which were sent from the
original form, so if you have a field called "ActorSearch" in your field, the WSProxy will also send
ActorSearch=foo to the remote service.
This is nice, but it's not always useful if you don't want to use Amazon's form field names, so I
can see why you might have wanted to use the different input name in your form. AFAIK, it's up to
the "sender" of the URL to make sure that request parameter names and values are URL-encoded, so I
am debating whether this might warrant an input module which performs URL encoding.
Thoughts?
Regards,
Tony
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
> > So,
> > how do I ensure that my web service request is being sent with "%20"
instead
> > of " " or "+"?
> >
> > Jeff,
>
> I've noticed this too, a while ago.... it is a little frustrating
> not being able to have it change
> the + to a %20... perhaps this type of encoding behavior needs to
> be added to the wsproxy.
It hadn't occured to me that this was a problem with wsproxy. I just thought
I needed to do something to convert the format. But then I realized that I
had to manually change my all my "&"s to "&"s. Which format should the
user be expected to use for the src attribute of the generator? It seems to
me the more human-readable form makes the most sense, with encoding
happening in the wsproxy.
> Anybody else have thoughts? What should the proper encoding of this be?
>
> Tony
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: wsproxy and request parameters
Posted by Tony Collen <co...@umn.edu>.
Jeff Ramsdale wrote:
> I discovered that changing my search from "johnny depp" to "depp" returns a
> list of results. The problem, then, would seem to be the encoding of spaces
> in my request parameters. When I submit my search form it converts the space
> to a "+" in my browser. Examining the logs I see that they show " ", not "+"
> on my failed searches. As mentioned before, in my sitemap if I change
> "{request-param:actor}" to "johnny%20depp" I also get a list of results. So,
> how do I ensure that my web service request is being sent with "%20" instead
> of " " or "+"?
Jeff,
I've noticed this too, a while ago.... it is a little frustrating not being able to have it change
the + to a %20... perhaps this type of encoding behavior needs to be added to the wsproxy.
Anybody else have thoughts? What should the proper encoding of this be?
Tony
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
RE: wsproxy and request parameters
Posted by Jeff Ramsdale <je...@earthlink.net>.
I discovered that changing my search from "johnny depp" to "depp" returns a
list of results. The problem, then, would seem to be the encoding of spaces
in my request parameters. When I submit my search form it converts the space
to a "+" in my browser. Examining the logs I see that they show " ", not "+"
on my failed searches. As mentioned before, in my sitemap if I change
"{request-param:actor}" to "johnny%20depp" I also get a list of results. So,
how do I ensure that my web service request is being sent with "%20" instead
of " " or "+"?
Thanks,
Jeff
> -----Original Message-----
> From: Jeff Ramsdale [mailto:jefe182@earthlink.net]
> Sent: Thursday, September 11, 2003 12:45 AM
> To: cocoon-users@apache.org
> Subject: wsproxy and request parameters
>
>
> I'm playing with the Amazon.com web service toolkit and Cocoon.
>
> Here's my sitemap snippete:
>
> <map:match pattern="doSearch.html">
> <map:generate name="wsproxy"
> src="http://xml.amazon.com/onca/xml3?t=webservices-20&dev-t=[my dev
> code]&ActorSearch={request-param:actor}&mode=dvd&type=
> lite&p
> age=1&f=xml" />
> <map:transform src="stylesheets/lite-data-to-html.xsl" />
> <map:serialize type="xhtml" />
> </map:match>
>
> When I request my page with
> http://localhost:8888/Amazon/doSearch.html?actor=johnny+depp I
> don't get any
> results back from Amazon. When I replace "{request-param:actor}" in my
> sitemap with "johnny%20depp" I get a bunch of results back.
>
> Any hints on why my parameters aren't being passed along? I'm on a pretty
> recent CVS checkout of Cocoon.
>
> Thanks,
>
> Jeff
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
> For additional commands, e-mail: users-help@cocoon.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org