OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

Hello.

I'm really sorry to bother people here, but I just found an 
inconsistency between MySpace OAuth and Java OAuth (and possibly Hi5 
too), I decided to ask the developers here.. since they should both be 
backed by Shindig.

It deals with generating the Base String with parameter names that have 
funky characters.

Looking over the Java OAuth code, it looks like they are double encoding 
names/values to generate the Base String.  While looking at the MySpace 
OAuth Tool, it looks like they are single encoding names, while double 
encoding values.  Though I could not find the Hi5 equivalent of the 
OAuth Tool, both MySpace and Hi5 are failing the signature test; so I'm 
just assuming it's the same issue.

For example; a parameter like this:

t:formdata=a:a

was converted into the base string by MySpace like this:

t%3Aformdata%3Da%253Aa



I hope this isn't annoying, I just wanted to get the consensus from all 
developers involved, on how this should work; so we can patch Shindig or 
patch Java OAuth.  ( since everyone is backed by Shindig, my guess it's 
actually Shindig's OAuth code ).

What is the expected behavior for funky characters in parameter names?

Re: OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

Thank you.  So.. anybody from MySpace here?  Because like I pointed out 
in my first email, it looks like the OAuthTool is not generating the 
Base String properly.. it looks like it's single encoding parameter 
names, while double encoding parameter values..

Does that use any code in Shindig?  Or any other OAuth library code?


Brian Eaton wrote:
>> I would look at it, but where is this "SigningFetcher" you speak of? :)
> 
> It's in Shindig, not java OAuth.
> 
> http://svn.apache.org/repos/asf/incubator/shindig/trunk/java/gadgets/src/main/java/org/apache/shindig/gadgets/SigningFetcher.java
> 
>> So you're saying that it should behave like Java OAuth?  Why don't you use Java OAuth in the first place?
> 
> We do.  Signed fetch is a concept invented for opensocial, built on
> top of the OAuth signature specification.  SigningFetcher adds
> opensocial specific stuff on top of the base OAuth java libs.
> 
> Cheers,
> Brian

Re: OAuth Question; double encode parameter names?

[John Panzer <jp...@google.com>]

Unicode delenda est!

On Fri, Apr 4, 2008 at 2:42 PM, Kevin Brown <et...@google.com> wrote:

> On Fri, Apr 4, 2008 at 2:24 PM, Brian Eaton <be...@google.com> wrote:
>
> > On Fri, Apr 4, 2008 at 2:20 PM, Kevin Brown <et...@google.com> wrote:
> > >  > Quick!  What happens if someone passes in
> "%ff%4fpensocial_owner_id"
> > >  > to our java code and then we pass it on to .NET code running in a
> > >  > japanese locale?
> > >
> > >
> > >  Er, shouldn't this always be UTF-8 encoded, in which case %ff isn't a
> > valid
> > >  character anyway? It seems to me that if we're decoding and
> re-encoding
> > >  anyway we'd eliminate most of these sorts of things.
> >
> > You're right, %ff%4f is unicode, there is a UTF-8 version we'd accept
> > though.
> >
> > Java does not canonicalize the halfwidth to ASCII equivalents.  Other
> > platforms do (I think for sort order?), so the disconnect causes
> > problems.
>
>
> This is why everyone should speak Latin.
>
>
> >
> >
> > Cheers,
> > Brian
> >
>
>
>
> --
> ~Kevin
>

Re: OAuth Question; double encode parameter names?

[Kevin Brown <et...@google.com>]

On Fri, Apr 4, 2008 at 2:24 PM, Brian Eaton <be...@google.com> wrote:

> On Fri, Apr 4, 2008 at 2:20 PM, Kevin Brown <et...@google.com> wrote:
> >  > Quick!  What happens if someone passes in "%ff%4fpensocial_owner_id"
> >  > to our java code and then we pass it on to .NET code running in a
> >  > japanese locale?
> >
> >
> >  Er, shouldn't this always be UTF-8 encoded, in which case %ff isn't a
> valid
> >  character anyway? It seems to me that if we're decoding and re-encoding
> >  anyway we'd eliminate most of these sorts of things.
>
> You're right, %ff%4f is unicode, there is a UTF-8 version we'd accept
> though.
>
> Java does not canonicalize the halfwidth to ASCII equivalents.  Other
> platforms do (I think for sort order?), so the disconnect causes
> problems.


This is why everyone should speak Latin.


>
>
> Cheers,
> Brian
>



-- 
~Kevin

Re: OAuth Question; double encode parameter names?

[Brian Eaton <be...@google.com>]

On Fri, Apr 4, 2008 at 2:20 PM, Kevin Brown <et...@google.com> wrote:
>  > Quick!  What happens if someone passes in "%ff%4fpensocial_owner_id"
>  > to our java code and then we pass it on to .NET code running in a
>  > japanese locale?
>
>
>  Er, shouldn't this always be UTF-8 encoded, in which case %ff isn't a valid
>  character anyway? It seems to me that if we're decoding and re-encoding
>  anyway we'd eliminate most of these sorts of things.

You're right, %ff%4f is unicode, there is a UTF-8 version we'd accept though.

Java does not canonicalize the halfwidth to ASCII equivalents.  Other
platforms do (I think for sort order?), so the disconnect causes
problems.

Cheers,
Brian

Re: OAuth Question; double encode parameter names?

[Kevin Brown <et...@google.com>]

On Fri, Apr 4, 2008 at 2:10 PM, Brian Eaton <be...@google.com> wrote:

> On Fri, Apr 4, 2008 at 1:43 PM, Fernando Padilla <fe...@alum.mit.edu>
> wrote:
> > I did the search, the docs were pretty obtuse though..
>
> Yeah.  Almost by definition you're dealing with the intersection of
> obscure behavior in two different codebases.  Nobody understands it,
> much less includes it in test plans.
>
> >  I still don't see the need to have this check here.. if the bug is that
> > someone passes in some %u encoded character and some cisco router
> doesn't
> > detect it properly..  I still don't see what Shindig has to do with
> that..
> > If you're worried about that, outlaw "%u" encoding.  Or decode all
> values
> > first, then re-encode them. (wait, we're already doing that).
>
> Quick!  What happens if someone passes in "%ff%4fpensocial_owner_id"
> to our java code and then we pass it on to .NET code running in a
> japanese locale?


Er, shouldn't this always be UTF-8 encoded, in which case %ff isn't a valid
character anyway? It seems to me that if we're decoding and re-encoding
anyway we'd eliminate most of these sorts of things.


>
>
> The answer depends on the intersection of obscure behavior in two
> different codebases.   0xff 0x4f is a halfwidth unicode 'o', and some
> systems canonicalize it to a normal ascii 'o' and others don't.  This
> kind of problem is why blacklisting invalid characters is a bad idea
> for security sensitive code.  Whitelist what you know is safe.  Reject
> everything else.
>
> >  Plus, having Shindig fix for some broken detection system, I think is
> > totally off-base..
>
> The reports of problems in IDS are an indication of the general type
> of attack.  The root cause of the problem is having different
> platforms (Java vs .NET vs Perl vs Python vs C++ vs who-knows-what)
> trying to interpret arbitrary sequences of bytes in the exact same
> way.  Because signed fetch involves our java code talking to arbitrary
> servers, we need to be concerned about the issue.  The simplest fix is
> to disallow arbitrary sequences of bytes.
>
> Cheers,
> Brian
>



-- 
~Kevin

Re: OAuth Question; double encode parameter names?

[Brian Eaton <be...@google.com>]

On Fri, Apr 4, 2008 at 1:43 PM, Fernando Padilla <fe...@alum.mit.edu> wrote:
> I did the search, the docs were pretty obtuse though..

Yeah.  Almost by definition you're dealing with the intersection of
obscure behavior in two different codebases.  Nobody understands it,
much less includes it in test plans.

>  I still don't see the need to have this check here.. if the bug is that
> someone passes in some %u encoded character and some cisco router doesn't
> detect it properly..  I still don't see what Shindig has to do with that..
> If you're worried about that, outlaw "%u" encoding.  Or decode all values
> first, then re-encode them. (wait, we're already doing that).

Quick!  What happens if someone passes in "%ff%4fpensocial_owner_id"
to our java code and then we pass it on to .NET code running in a
japanese locale?

The answer depends on the intersection of obscure behavior in two
different codebases.   0xff 0x4f is a halfwidth unicode 'o', and some
systems canonicalize it to a normal ascii 'o' and others don't.  This
kind of problem is why blacklisting invalid characters is a bad idea
for security sensitive code.  Whitelist what you know is safe.  Reject
everything else.

>  Plus, having Shindig fix for some broken detection system, I think is
> totally off-base..

The reports of problems in IDS are an indication of the general type
of attack.  The root cause of the problem is having different
platforms (Java vs .NET vs Perl vs Python vs C++ vs who-knows-what)
trying to interpret arbitrary sequences of bytes in the exact same
way.  Because signed fetch involves our java code talking to arbitrary
servers, we need to be concerned about the issue.  The simplest fix is
to disallow arbitrary sequences of bytes.

Cheers,
Brian

Re: OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

I did the search, the docs were pretty obtuse though..

I still don't see the need to have this check here.. if the bug is that 
someone passes in some %u encoded character and some cisco router 
doesn't detect it properly..  I still don't see what Shindig has to do 
with that..  If you're worried about that, outlaw "%u" encoding.  Or 
decode all values first, then re-encode them. (wait, we're already doing 
that).

Plus they are just referring to "exploit signatures", so you can use %u 
in either parameter names or parameter values.. it still changes the 
signature used by some stupid detection system (really what detection 
system looks for a precise string match on a url anyhow).

Plus, having Shindig fix for some broken detection system, I think is 
totally off-base..


But then again, this is just the first time I've heard of this. :)

And for now I'm just happy with supporting ":".

So I won't push anymore.. unless someone else agrees with me. :)

I just created a jira issue, with patch.



Brian Eaton wrote:
> On Fri, Apr 4, 2008 at 12:32 PM, Fernando Padilla <fe...@alum.mit.edu> wrote:
>>  Is there any real reason to restrict the parameter names beyond
>> oauth/opensocial?  Should I just submit the patch to remove the
>> ALLOWE_PARAM_NAME checking all together?
> 
> The check is there for security reasons, to prevent any attempt to
> smuggle in an opensocial param that our code misses but the verifying
> code accepts.  Search the web for "IDS bypass" for pointers to various
> kinds of attacks that might be possible.
> 
> Modifying the regex to allow the ":" character as well as the
> characters it permits already seems reasonable, but we should be
> paranoid about adding additional characters.
> 
> Cheers,
> Brian

Re: OAuth Question; double encode parameter names?

[Brian Eaton <be...@google.com>]

On Fri, Apr 4, 2008 at 12:32 PM, Fernando Padilla <fe...@alum.mit.edu> wrote:
>  Is there any real reason to restrict the parameter names beyond
> oauth/opensocial?  Should I just submit the patch to remove the
> ALLOWE_PARAM_NAME checking all together?

The check is there for security reasons, to prevent any attempt to
smuggle in an opensocial param that our code misses but the verifying
code accepts.  Search the web for "IDS bypass" for pointers to various
kinds of attacks that might be possible.

Modifying the regex to allow the ":" character as well as the
characters it permits already seems reasonable, but we should be
paranoid about adding additional characters.

Cheers,
Brian

Re: OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

Dang I totally flaked on that one. Sorry. :)  Thank you for the link..

So SigningFetcher is making sure that a makeRequest doesn't try to pass 
in any special oauth/opensocial parameters:

    return (!(canonParamName.startsWith("oauth") ||
         canonParamName.startsWith("xoauth") ||
         canonParamName.startsWith("opensocial")) &&

But then it goes extra anal and doesn't allow any parameters that it 
doesn't like:

         ALLOWED_PARAM_NAME.matcher(canonParamName).matches());

where:

protected static final Pattern ALLOWED_PARAM_NAME = 
Pattern.compile("[\\w_\\-]+");


Is there any real reason to restrict the parameter names beyond 
oauth/opensocial?  Should I just submit the patch to remove the 
ALLOWE_PARAM_NAME checking all together?




Brian Eaton wrote:
>> I would look at it, but where is this "SigningFetcher" you speak of? :)
> 
> It's in Shindig, not java OAuth.
> 
> http://svn.apache.org/repos/asf/incubator/shindig/trunk/java/gadgets/src/main/java/org/apache/shindig/gadgets/SigningFetcher.java
> 
>> So you're saying that it should behave like Java OAuth?  Why don't you use Java OAuth in the first place?
> 
> We do.  Signed fetch is a concept invented for opensocial, built on
> top of the OAuth signature specification.  SigningFetcher adds
> opensocial specific stuff on top of the base OAuth java libs.
> 
> Cheers,
> Brian

Re: OAuth Question; double encode parameter names?

[Brian Eaton <be...@google.com>]

> I would look at it, but where is this "SigningFetcher" you speak of? :)

It's in Shindig, not java OAuth.

http://svn.apache.org/repos/asf/incubator/shindig/trunk/java/gadgets/src/main/java/org/apache/shindig/gadgets/SigningFetcher.java

> So you're saying that it should behave like Java OAuth?  Why don't you use Java OAuth in the first place?

We do.  Signed fetch is a concept invented for opensocial, built on
top of the OAuth signature specification.  SigningFetcher adds
opensocial specific stuff on top of the base OAuth java libs.

Cheers,
Brian

Re: OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

So you're saying that it should behave like Java OAuth?  Why don't you 
use Java OAuth in the first place?

Brian Eaton wrote:
> On Fri, Apr 4, 2008 at 8:48 AM, Fernando Padilla <fe...@alum.mit.edu> wrote:
>>  For example; a parameter like this:
>>
>>  t:formdata=a:a
>>
>>  was converted into the base string by MySpace like this:
>>
>>  t%3Aformdata%3Da%253Aa
>>
>>  I hope this isn't annoying, I just wanted to get the consensus from all
>> developers involved, on how this should work; so we can patch Shindig or
>> patch Java OAuth.  ( since everyone is backed by Shindig, my guess it's
>> actually Shindig's OAuth code ).
> 
> Great guess.  The bug is in SigningFetcher.java, in the function
> allowParam.  Shindig is aggressive about looking for weird characters
> in parameter names, to avoid cases where people try to sneak things
> like %6fpensocial_owner_id into signed fetch requests.
> 
> We can probably make the regex a little bit more liberal to allow the
> ':' character.  Want to prepare a patch?
> 
> Cheers,
> Brian

Re: OAuth Question; double encode parameter names?

[Fernando Padilla <fe...@alum.mit.edu>]

I would look at it, but where is this "SigningFetcher" you speak of? :)

I checked out: http://oauth.googlecode.com/svn/code/java

but it's not there.


Brian Eaton wrote:
> On Fri, Apr 4, 2008 at 8:48 AM, Fernando Padilla <fe...@alum.mit.edu> wrote:
>>  For example; a parameter like this:
>>
>>  t:formdata=a:a
>>
>>  was converted into the base string by MySpace like this:
>>
>>  t%3Aformdata%3Da%253Aa
>>
>>  I hope this isn't annoying, I just wanted to get the consensus from all
>> developers involved, on how this should work; so we can patch Shindig or
>> patch Java OAuth.  ( since everyone is backed by Shindig, my guess it's
>> actually Shindig's OAuth code ).
> 
> Great guess.  The bug is in SigningFetcher.java, in the function
> allowParam.  Shindig is aggressive about looking for weird characters
> in parameter names, to avoid cases where people try to sneak things
> like %6fpensocial_owner_id into signed fetch requests.
> 
> We can probably make the regex a little bit more liberal to allow the
> ':' character.  Want to prepare a patch?
> 
> Cheers,
> Brian

Re: OAuth Question; double encode parameter names?

[Brian Eaton <be...@google.com>]

On Fri, Apr 4, 2008 at 8:48 AM, Fernando Padilla <fe...@alum.mit.edu> wrote:
>  For example; a parameter like this:
>
>  t:formdata=a:a
>
>  was converted into the base string by MySpace like this:
>
>  t%3Aformdata%3Da%253Aa
>
>  I hope this isn't annoying, I just wanted to get the consensus from all
> developers involved, on how this should work; so we can patch Shindig or
> patch Java OAuth.  ( since everyone is backed by Shindig, my guess it's
> actually Shindig's OAuth code ).

Great guess.  The bug is in SigningFetcher.java, in the function
allowParam.  Shindig is aggressive about looking for weird characters
in parameter names, to avoid cases where people try to sneak things
like %6fpensocial_owner_id into signed fetch requests.

We can probably make the regex a little bit more liberal to allow the
':' character.  Want to prepare a patch?

Cheers,
Brian