You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "guohao-rosicky (via GitHub)" <gi...@apache.org> on 2023/07/06 03:58:51 UTC
[GitHub] [ozone] guohao-rosicky opened a new pull request, #5026: HDDS-8965. Stream multi part upload support on gateway
guohao-rosicky opened a new pull request, #5026:
URL: https://github.com/apache/ozone/pull/5026
## What changes were proposed in this pull request?
Stream multi part upload support on gateway
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8965
## How was this patch tested?
ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628827860
> If stream is used in the test, is it enough to change it in the config file?
> `hadoop-ozone/integration-test/src/test/resources/ozone-site.xml`
That file contains settings applied to all integration tests (tests in `hadoop-ozone/integration-test`). There are similar files for tests in some other modules:
```
hadoop-hdds/framework/src/test/resources/ozone-site.xml
hadoop-hdds/container-service/src/test/resources/ozone-site.xml
```
You can also apply specific settings in any test class by making the config changes before the test, e.g.:
https://github.com/apache/ozone/blob/e6028b79ec7224c935b2e2219606e6b3d870c5df/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestListStatus.java#L68-L75
S3 Gateway is not created in `MiniOzoneCluster`, hence not tested in integration tests. It can be tested end-to-end in [acceptance tests](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/README.md#tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ivandika3 commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1630189216
@guohao-rosicky Thank you for the detailed explanation.
> The stream port has a default value that is incorrect when a client connects to the stream server.
Regarding this, is it possible to return the default port values (e.g. `DFS_CONTAINER_RATIS_DATASTREAM_PORT_DEFAULT`) instead of returning the RATIS portDetails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1630062168
> `TestPartUploadWithStream` requires datastream to be enabled. It sets the config in its `@BeforeClass` method.
Thanks @adoroszlai Let me change it and try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1635328524
hi @szetszwo . It has been modified and the ci has been passed, can you review it for me, thanks!
see:
https://github.com/guohao-rosicky/ozone/actions/runs/5550287644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1622937451
```
<property>
<name>dfs.container.ratis.datastream.enabled</name>
<value>true</value>
</property>
```
Change the configuration to enable, because the pipeline created by scm will not enable the datastream port if it is not enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628498488
The client needs to access the port of the stream service when communicating with the stream server.
This port is provided in the DatanodeDetails provided by the datanode report. If the stream service is not enabled on the datanode, there will be no stream port in the DatanodeDetails.
See:
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis#start
https://github.com/apache/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java#L530
The stream port has a default value that is incorrect when a client connects to the stream server.
See:
org.apache.hadoop.hdds.protocol.DatanodeDetails#getPort
https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java#L319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1623423830
> ```
> <property>
> <name>dfs.container.ratis.datastream.enabled</name>
> <value>true</value>
> </property>
> ```
>
> Change the configuration to enable, because the pipeline created by scm will not enable the datastream port if it is not enabled
I don't think the default setting should be changed for this. S3 Gateway should be able to handle both settings. Changing the default might just hide some issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1629033206
Thanks @adoroszlai , as you said, I modified the `hadoop-ozone/integration-test/src/test/resources/ozone-site.xml`
and changed the default configuration(ozone-default.xml) back and the CI has been passed .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1629067862
Thanks @guohao-rosicky. If I understand correctly, only `TestPartUploadWithStream` requires datastream to be enabled. It sets the config in its `@BeforeClass` method. Since it is not an integration test, the change in `hadoop-ozone/integration-test/src/test/resources/ozone-site.xml` does not affect it and is not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628105236
> I don't think the default setting should be changed for this. S3 Gateway should be able to handle both settings. Changing the default might just hide some issues.
The default scm creates a pipeline, and if this configuration is false, it will not specify the correct stream port, causing the gateway to connect to the wrong stream port on the pipeline, and the communication will be abnormal.
@adoroszlai @szetszwo Should I first change the configuration of the stream port when creating the pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1634038985
> ... Let me change it and try it.
@guohao-rosicky , Are you going to try enabling streaming using `@BeforeClass`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628103995
The default scm creates a pipeline, and if this configuration is false, it will not specify the correct stream port, causing the gateway to connect to the wrong stream port on the pipeline, and the communication will be abnormal.
@adoroszlai @szetszwo Should I first change the configuration of the stream port when creating the pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo merged pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #5026:
URL: https://github.com/apache/ozone/pull/5026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ivandika3 commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628446077
Hi @guohao-rosicky, thank you for working on this. I'm interested in this feature as well.
> The default scm creates a pipeline, and if this configuration is false, it will not specify the correct stream port, causing the gateway to connect to the wrong stream port on the pipeline, and the communication will be abnormal.
Could you help clarify what do you mean by this (e.g. which class/implementation has this behavior)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1630297203
> Regarding this, is it possible to return the default port values (e.g. `DFS_CONTAINER_RATIS_DATASTREAM_PORT_DEFAULT`) instead of returning the RATIS portDetails?
@ivandika3
Yes, the port issue. I'm following up on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #5026: HDDS-8965. Stream multi part upload support on gateway
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5026:
URL: https://github.com/apache/ozone/pull/5026#issuecomment-1628799879
The [CI](https://github.com/guohao-rosicky/ozone/actions/runs/5506143442) passed, PTAL @szetszwo thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org