You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Marton Elek (Jira)" <ji...@apache.org> on 2021/03/09 19:56:00 UTC
[jira] [Commented] (HDDS-4856) Ruby S3 SDK never get authenticated
by Ozone
[ https://issues.apache.org/jira/browse/HDDS-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298298#comment-17298298 ]
Marton Elek commented on HDDS-4856:
-----------------------------------
Thank you very much the detailed problem description [~kuenishi], it helped a lot.
I did debug it locally, and it seems to be the limitation of Jersey REST framework (the jetty issue is closed with the same conclusion).
I think it's not a kind thing to send an empty Content-Type ;-) But if AWS supports it we can do the same: As a workaround I created a simple web filter which replaces the Content-Type with null if it's an empty string. Tested locally, and it seems to be working (thanks a lot for the ruby scripts).
PR is opened, please test if you have time...
https://github.com/apache/ozone/pull/2013
> Ruby S3 SDK never get authenticated by Ozone
> --------------------------------------------
>
> Key: HDDS-4856
> URL: https://issues.apache.org/jira/browse/HDDS-4856
> Project: Apache Ozone
> Issue Type: Bug
> Components: S3
> Affects Versions: 1.0.0
> Environment: Secure setup of Ozone 1.0.0
> Reporter: UENISHI Kota
> Assignee: Marton Elek
> Priority: Major
> Labels: pull-request-available
> Attachments: ozone-test.py, ozone-test.rb, ruby-sdk-patch.diff
>
>
> When the very first call by Ruby client against secure setup of Ozone, the server returns 400 no matter how valid the request is. See the attached ruby-sdk-patch.diff, which adds some tests on S3 auth header signature-to-sign generation. It consists of two test additions, the "2" is the one generated by boto3, the "3" is generated by aws-ruby-sdk. Both passes the additional tests, which are definitely valid.
> However, when real HTTP request is sent by Ruby client, e.g. ozone-test.rb attached, it fails with 400. The header was like this (though the host names and domains are masked):
> {quote}GET //ozone.example.com:9879/sandbox?list-type=2&max-keys=1 HTTP/1.1
> Content-Type:
> Accept-Encoding:
> User-Agent: aws-sdk-ruby3/3.112.0 ruby/2.7.2 x86_64-linux aws-sdk-s3/1.88.1
> Host: ozone.example.com:9879
> X-Amz-Date: 20210222T110554Z
> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> Authorization: AWS4-HMAC-SHA256 Credential=kota@EXAMPLE.COM/20210222/foobar/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0c9469f018f5
> b3fd2cff6f8d4e4963f50aa71c6704def59527634404f5fc98a9
> Content-Length: 0
> Accept: */*{quote}
> On the other hand, request headers made by boto3 was:
> {quote}GET //ozone.example.com:9879/sandbox?list-type=2&encoding-type=url HTTP/1.1
> Host: ozone.example.com:9879
> Accept-Encoding: identity
> User-Agent: Boto3/1.17.12 Python/3.9.1 Linux/5.10.14-arch1-1 Botocore/1.20.12
> X-Amz-Date: 20210222T110829Z
> X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> Authorization: AWS4-HMAC-SHA256 Credential=kota@EXAMPLE.COM/20210222/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=94302f21cccac8832d3e
> 4fe25c5f6d8a0307188fb0e1b1983264339381d21dac{quote}
> The difference of these requests are IMHO, "Content-Type" and "Accept-Encoding" are both empty in Ruby SDK. I'm afraid this error stems from partly Ruby SDK and partly from [Jetty Issue|https://github.com/eclipse/jetty.project/issues/2883]. The former sends empty header lines and the latter rejects them.
> And the s3g debug log (only error'ish part) follows:
> {quote}2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: chain=NoCacheFilter@5e600dd5==org.apache.hadoop.hdds.server.http.NoCacheFilter,inst=true,async=true-
> >safety@63a12c68==org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter,inst=true,async=true->info-page-redirect@576d5deb==org.apache.hadoop.ozone.s3.RootPageDis
> playFilter,inst=true,async=false->jaxrs@603a422==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=1,inst=true,async=false
> 2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter NoCacheFilter@5e600dd5==org.apache.hadoop.hdds.server.http.NoCacheFilter,inst=true,async
> =true
> 2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter safety@63a12c68==org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter,inst=
> true,async=true
> 2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call filter info-page-redirect@576d5deb==org.apache.hadoop.ozone.s3.RootPageDisplayFilter,inst=true,
> async=false
> 2021-02-22 20:55:54,450 [qtp1637061418-81] DEBUG servlet.ServletHandler: call servlet jaxrs@603a422==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=1,inst=true
> ,async=false
> 2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: sendError HttpChannelState@4893b376{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=tru
> e al=0}
> 2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.session: Leaving scope org.eclipse.jetty.server.session.SessionHandler367746789==dftMaxIdleSec=-1 dispatch=REQUEST, a
> sync=false, session=null, oldsession=null, oldsessionhandler=null
> 2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.Server: handled=true async=false committed=true on HttpChannelOverHttp@769bb34b{s=HttpChannelState@4893b376{s=HANDLIN
> G rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true al=0},r=1,c=false/false,a=HANDLING,uri=https://ozone.example.com:9879/sandbox?list-type=2&ma
> x-keys=1,age=2}
> 2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: unhandle HttpChannelState@4893b376{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=true i=true
> al=0}
> 2021-02-22 20:55:54,451 [qtp1637061418-81] DEBUG server.HttpChannelState: nextAction(false) SEND_ERROR HttpChannelState@4893b376{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=f
> alse se=false i=false al=0}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org