You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2025/05/07 01:53:00 UTC
[jira] [Assigned] (HDDS-12123) PutObject using the latest awscli later than or equal to 1.37.0 is fail
[ https://issues.apache.org/jira/browse/HDDS-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Andika reassigned HDDS-12123:
----------------------------------
Assignee: Ivan Andika
> PutObject using the latest awscli later than or equal to 1.37.0 is fail
> -----------------------------------------------------------------------
>
> Key: HDDS-12123
> URL: https://issues.apache.org/jira/browse/HDDS-12123
> Project: Apache Ozone
> Issue Type: Bug
> Components: s3gateway
> Affects Versions: 1.4.0, 1.4.1
> Environment: * awscli >= 1.37.0 (aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3)
> * Ozone 1.4.1 [compatible version|https://github.com/pfnet/ozone/commits/pfn-ozone-1.4.990101/] (based on ozone-1.4.1 tag + our internal patches)
> * Secure mode is enabled w/ Kerberos
> * w/o Ranger
> * ozone.http.policy = HTTP_AND_HTTPS
> Reporter: Kohei Sugihara
> Assignee: Ivan Andika
> Priority: Major
>
> S3G returns internal server errors (500) when we upload some files using awscli >= 1.37.0 (not AWS CLI version 2), the latest version of AWS CLI released in January 2025.
> {code:java}
> 2025-01-22 14:31:36,809 [qtp1327871893-36] WARN org.eclipse.jetty.server.HttpChannel: /ksugihara/bin-1m.dat
> javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IllegalStateException: Expected: 0 and actual 1048622 write sizes do not match
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
> at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> This happened on the S3 endpoint enabled with the secure mode. To explore this issue in more detail, we changed the ozone. HTTP.policy to HTTP_AND_HTTPS to check both endpoints. Curiously, this error does not occur on the HTTP endpoint but only on the HTTPS endpoint (port: 9879) in S3G. OM and SCM did not output failure logs in their stdout or audit logs, so I think this issue is closed within the S3G component.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred (500) when calling the PutObject operation (reached max retries: 4): Internal Server Error {code}
> Once downgrading the awscli package earlier than 1.37.0, the bug seems not occurs.
> {code:java}
> % pip3 install --upgrade 'awscli<1.37.0'
> % aws --version
> aws-cli/1.36.40 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.35.99
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat{code}
> Our current workarounds for using S3 endpoint:
> * Use earlier than awscli 1.37.0: pip3 install 'awscli<1.37.0'
> * Use the HTTP endpoint instead of the HTTPS endpoint
> I don't make sure this information is relavant to identify the root cause, awscli v1.37.x is also failure for file uploading to Apache Ozone 1.4{+}.0{+} cluster, but in this case S3G returns a conflict error when file is being overwritten. New upload that does not overwrite existing files seems to success.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp s3://ksugihara/bin-1m.dat .
> # overwrite upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred (Conflict) when calling the PutObject operation: Cannot overwrite file with directory
> # new upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ --profile pfs0 cp ./bin-1m.dat s3://ksugihara/bin-1m.dat.1
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat.1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org