You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/09/27 11:37:46 UTC
[GitHub] [ozone] captainzmc opened a new pull request #2682: HDDS-5705. [Ozone-Streaming] Streaming supports writing in Pipline mode
captainzmc opened a new pull request #2682:
URL: https://github.com/apache/ozone/pull/2682
## What changes were proposed in this pull request?
Streaming supports writing in Pipline mode
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-929809305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc removed a comment on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc removed a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-927810977
Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. After debug I found that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
I haven't found the cause of the problem so far. Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc removed a comment on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc removed a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-927810977
Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. After debug I found that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
I haven't found the cause of the problem so far. Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928924289
@szetszwo Please ignore the problem I asked earlier, I have found the root cause. I will fix the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #2682: HDDS-5705. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-927810977
Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. After debug I found that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
I haven't found the cause of the problem so far. Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo merged pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo merged pull request #2682:
URL: https://github.com/apache/ozone/pull/2682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#discussion_r717292689
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +186,47 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+// .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeer primary = null;
+ List<RaftPeer> raftPeers = new ArrayList();
Review comment:
RoutingTable only needs RaftPeerId. So we can have `List<RaftPeerId>` instead of `List<RaftPeer>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928924289
> Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
>
> But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. After debug I found that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
>
> I haven't found the cause of the problem so far. Do you have any suggestions?
>
> 
>
> 
@szetszwo Please ignore the problem, I have found the root cause. I will fix the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-929809305
> @captainzmc , can you add some description on the jira for this? The patch otherwise looks good. Please remove the commented out code.
Yes, I have updated jIRA and add the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928924289
> Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
>
> But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. After debug I found that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
>
> I haven't found the cause of the problem so far. Do you have any suggestions?
>
> 
>
> 
@szetszwo Please ignore the problem, I have found the root cause. I will fix the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-929298611
@captainzmc , can you add some description on the jira for this? The patch otherwise looks good. Please remove the commented out code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5705. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-927810977
Hi @szetszwo I have a question here. As far as I know if we support writing in pipline, we just need to build the right RoutingTable on the client side.
But after I pass in the RoutingTable, the pirmary block file can be init normally(call stack see the first image). However, the block file of peers cannot be properly init. I debug that the stream(RaftClientRequest request) method in ContainerStateMachine of peers is not called(see the call stack in the second image). The initialization request calls the applyTransaction(TransactionContext trx) method in ContainerStateMachine. The block file cannot be initialized because the stream method will not be called.
I haven't found the cause of the problem so far. Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928924289
@szetszwo Please ignore the problem I asked earlier, I have found the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#discussion_r718225475
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
+
+ for (DatanodeDetails dn : pipeline.getNodes()) {
+ RaftPeer.Builder raftPeerBuilder = RaftPeer.newBuilder();
+ raftPeerBuilder.setId(dn.getUuidString()).setAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_SERVER)
+ .getValue());
+ raftPeerBuilder.setDataStreamAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_DATASTREAM)
+ .getValue());
+ raftPeerBuilder.setAdminAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_ADMIN).getValue());
+ raftPeerBuilder.setClientAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS).getValue());
+ try {
+ if (dn == pipeline.getFirstNode()) {
+ primary = raftPeerBuilder.build().getId();
+
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
It should return null when there is an exception.
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
+
+ for (DatanodeDetails dn : pipeline.getNodes()) {
+ RaftPeer.Builder raftPeerBuilder = RaftPeer.newBuilder();
+ raftPeerBuilder.setId(dn.getUuidString()).setAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_SERVER)
+ .getValue());
+ raftPeerBuilder.setDataStreamAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_DATASTREAM)
+ .getValue());
+ raftPeerBuilder.setAdminAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_ADMIN).getValue());
+ raftPeerBuilder.setClientAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS).getValue());
Review comment:
Since RaftPeer is not needed, let's just create a RaftPeerId.
```
final RaftPeerId id = RaftPeerId.valueOf(dn.getUuidString());
```
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
Review comment:
It should have an "<>", i.e. `new ArrayList<>()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-929873710
hi @szetszwo, The code has been updated and tested. Can you help take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-929298611
@captainzmc , can you add some description on the jira for this? The patch otherwise looks good. Please remove the commented out code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928932476
> ... I have found the root cause.
That's great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-930244306
[CI is performing well on my personal branch](https://github.com/captainzmc/hadoop-ozone/runs/3743464804). This failure should be accidental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#discussion_r718225475
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
+
+ for (DatanodeDetails dn : pipeline.getNodes()) {
+ RaftPeer.Builder raftPeerBuilder = RaftPeer.newBuilder();
+ raftPeerBuilder.setId(dn.getUuidString()).setAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_SERVER)
+ .getValue());
+ raftPeerBuilder.setDataStreamAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_DATASTREAM)
+ .getValue());
+ raftPeerBuilder.setAdminAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_ADMIN).getValue());
+ raftPeerBuilder.setClientAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS).getValue());
+ try {
+ if (dn == pipeline.getFirstNode()) {
+ primary = raftPeerBuilder.build().getId();
+
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
It should return null when there is an exception.
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
+
+ for (DatanodeDetails dn : pipeline.getNodes()) {
+ RaftPeer.Builder raftPeerBuilder = RaftPeer.newBuilder();
+ raftPeerBuilder.setId(dn.getUuidString()).setAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_SERVER)
+ .getValue());
+ raftPeerBuilder.setDataStreamAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_DATASTREAM)
+ .getValue());
+ raftPeerBuilder.setAdminAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_ADMIN).getValue());
+ raftPeerBuilder.setClientAddress(dn.getIpAddress()
+ + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS).getValue());
Review comment:
Since RaftPeer is not needed, let's just create a RaftPeerId.
```
final RaftPeerId id = RaftPeerId.valueOf(dn.getUuidString());
```
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +187,48 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),
+ getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeerId primary = null;
+ List<RaftPeerId> raftPeers = new ArrayList();
Review comment:
It should have an "<>", i.e. `new ArrayList<>()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#discussion_r717292689
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +186,47 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+// .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeer primary = null;
+ List<RaftPeer> raftPeers = new ArrayList();
Review comment:
RoutingTable only needs RaftPeerId. So we can have List<RaftPeerId> instead of List<RaftPeer>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928932476
> ... I have found the root cause.
That's great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#issuecomment-928924289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2682: HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2682:
URL: https://github.com/apache/ozone/pull/2682#discussion_r717292689
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +186,47 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+// .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeer primary = null;
+ List<RaftPeer> raftPeers = new ArrayList();
Review comment:
RoutingTable only needs RaftPeerId. So we can have List<RaftPeerId> instead of List<RaftPeer>.
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -184,7 +186,47 @@ private DataStreamOutput setupStream() throws IOException {
ContainerCommandRequestMessage.toMessage(builder.build(), null);
return Preconditions.checkNotNull(xceiverClient.getDataStreamApi())
- .stream(message.getContent().asReadOnlyByteBuffer());
+// .stream(message.getContent().asReadOnlyByteBuffer());
+ .stream(message.getContent().asReadOnlyByteBuffer(),getRoutingTable(pipeline));
+ }
+
+ public RoutingTable getRoutingTable(Pipeline pipeline) {
+ RaftPeer primary = null;
+ List<RaftPeer> raftPeers = new ArrayList();
Review comment:
RoutingTable only needs RaftPeerId. So we can have `List<RaftPeerId>` instead of `List<RaftPeer>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org